http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 --- Comment #1 from Jacek Ablewicz <abl@biblos.pk.edu.pl> --- (In reply to Magnus Enger from comment #0)

A quick and dirty fix can be found here:

Works for me (3.14.5, apache + mod_perl). Or at least it seems to - hard to tell for sure, as this nasty bug does recur in semi-random fashion. Sometimes the 1st failed search occurs after a couple of minutes after server restart, and sometimes it works seemingly fine for hours.. I got an impression that as long as all processes/workers are relativelly busy, this issue is much harder to reproduce. Also it seems to happen somehow more often / earlier in plack + starman test configuration (or maybe it is just a coincidence). This Q&D fix is kind of ugly, though ;). It does quickly result in a lot of accumulated zebra processes (never to be used again, with postponed death sentences). Wouldn't it be better to destroy the Zconn connection as soon as possible when it is no longer needed? Sadly, ZOOM::Connection object does not automatically disconnect on DESTROY() like DBI does; I guess some kind of wrapper class may be needed to solve this. Also I wonder what kind of performance impact this (= not caching zebra connections at all) may have in plain CGI installations. Perhaps some plack-related fixes in the code should be controlled by new configuration setting (preferably in koha-conf.xml) which will indicate that system is, or is not running in persistent environment (and what kind: plack/*, mod_perl/persistent, mod_perl/semi-persistent ..). Btw, bug 11701 (pushed yesterday) got me thinking: looks like Zconn caching was not plack compatible at all. May it be somehow related to this bug (especially sync vs async thing)? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 Liz Rea <liz@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |liz@catalyst.net.nz --- Comment #3 from Liz Rea <liz@catalyst.net.nz> --- This is definitely a problem, we have seen this. Our solution was to use a zconn per search. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com --- Comment #5 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Chris Cormack from comment #4)

I can do a patch for this, which makes us use a new zebra connection each time. Or I can do a patch which would build a dependency on yaz-proxy so that we use that instead.

The first is obviously a lot faster, and is what zebra is designed for, it is not built for persistent connections, the second would allow us to pool connections, but would add a new software dependency

Opinions?

I'd go for the approach that creates a new connection. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 --- Comment #6 from Chris Cormack <chris@bigballofwax.co.nz> --- Created attachment 32102 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32102&action=edit Bug 12005 : Creating a new zebra connection for each time we need one Zebra is not designed to have persistent connections, under cgi this didn't matter the scripts would get a new connection each time, but under plack we try to use dead connections This patch changes it so plack works the same way that cgi did. To test: Apply this patch Do some searches Check everything still works -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 --- Comment #8 from Chris Cormack <chris@bigballofwax.co.nz> --- (In reply to Jacek Ablewicz from comment #7)

(In reply to Chris Cormack from comment #6)

...

This patch changes it so plack works the same way that cgi did.

At the moment I don't have a persistent install properly set up & suitable for any meaningfull testing, but - at least at the 1st glance, this proposed patch doesn't look quite right to me:

1) Completely abandoninig Zconn caching would have some measurable impact (performance-wise and memory-footprint-wise) in non-persistent installations. How significant such an impact may be in the practical circumstances is another question - while Koha does not seem to use cached Zconn very often, in those rare (?) cases when it does, cached Zconn is being used rather extensivelly (eg. when performing authority searches).

It would if it was working, but the caching has never worked. It has never been doing caching, you make a connection, and use it and then cgi dies and the connection disappears. In plack you make a connection, you use it, you stop using it, you asking for a new connection, it tries to give you back the old one that is dead. Searches break. So while I agree caching would probably be good, the fact is there is no performance hit from removing it, because we never had it.

2) Removing a code part which used to destroy() previously made zebra connection may be not such a good idea IMO. AFAIRC (it's been several months since I had a look at that code), previously estabilished Zconn is not being atomatically destroyed when the variable which holds it gets udefined, goes out of scope, etc. Unless I'm very much mistaken - with this patch applied, when running in persistent environment - yes, Koha will indeed make brand new zebra connection each time it does perform a search, but the problem is that previously made connection[s] would never get destroyed (and they will accumulate pretty quicky, eventually leading to the crash when the system goes out of available RAM for new zebra processes, free filehandles for subsequent connections and so on).

THis is not our experience we have been running it live in production for 6 months, zebra kills the connections without the destroy -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #32102|0 |1 is obsolete| | --- Comment #9 from Paul Poulain <paul.poulain@biblibre.com> --- Created attachment 32194 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32194&action=edit Bug 12005 : Creating a new zebra connection for each time we need one Zebra is not designed to have persistent connections, under cgi this didn't matter the scripts would get a new connection each time, but under plack we try to use dead connections This patch changes it so plack works the same way that cgi did. To test: Apply this patch Do some searches Check everything still works Signed-off-by: Paul Poulain <paul.poulain@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 --- Comment #12 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 32221 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32221&action=edit Bug 12005: Remove useless parameters C4::Context::Zconn subroutine takes too much more paramters. This subroutine is never called with more that 2 parameters. Others can be removed. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 --- Comment #14 from Jacek Ablewicz <abl@biblos.pk.edu.pl> --- CGI searches do seemingly work, but what I'm getting in my tests (clean master + bug 12005) still suggests that zebra connections have a tendency to accumulate with this patches applied. E.g., opac-authorities-home.pl script does create 21 of them, and they are getting destroyed all together only when the script ends. I dunno - maybe this only happens for a specific zebra / yaz / ZOOM versions, or depends on some configuration settings? On my test rig I have: zebra: 2.0.44-3 yaz: 4.2.30-2 libnet-z3950-zoom-perl 1.26-1+b2 They are all pretty old, but I guess it may be not that uncommon config (such versions are shipped with debian wheezy AFAIRC). In test database I have authorities linked to biblio records by using $9 subfields / auth record IDs. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 --- Comment #16 from Chris Cormack <chris@bigballofwax.co.nz> --- (In reply to Jacek Ablewicz from comment #15)

Also, misc/link_bibs_to_authorities.pl script doesn't work properly any longer:

misc/link_bibs_to_authorities.pl --test

after 20 or so seconds, it's starting to throw the following errors:

oAuth error: Connect failed (10000) unix:/home/koha/koha-dev/var/run/zebradb/authoritysocket ZOOM

and zebra daemons count climbs to (and stays at) 1019.

Ah yes, that looks like a bug in the script, in that it is making a new connection every query, instead of only once. Ill have a look at fixing that script. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 --- Comment #18 from Jacek Ablewicz <abl@biblos.pk.edu.pl> --- BTW, regarding the performance hit (after ditching Zconn caching) - I did some additional tests (with both current patch set and the Q&D patch from comment #0 somehow combined, so the old/previous connections are destroyed manually before making new ones): OPAC and staff search performaces are IMO not being affected in any signifficant way, but that particlar script (link_bibs_to_authorities.pl) is another story: without Zconn caching, it runs ca 3x slower.. However, it's probably not something that is being run every day (?) so maybe it's not such a big problem? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 --- Comment #20 from Chris Cormack <chris@bigballofwax.co.nz> --- Working on follow up patches for those 2 scripts now -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #32218|0 |1 is obsolete| | Attachment #32221|0 |1 is obsolete| | --- Comment #21 from Chris Cormack <chris@bigballofwax.co.nz> --- Created attachment 32334 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32334&action=edit Bug 12005 : Creating a new zebra connection for each time we need one Zebra is not designed to have persistent connections, under cgi this didn't matter the scripts would get a new connection each time, but under plack we try to use dead connections This patch changes it so plack works the same way that cgi did. To test: Apply this patch Do some searches Check everything still works -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 --- Comment #23 from Jacek Ablewicz <abl@biblos.pk.edu.pl> --- (In reply to Chris Cormack from comment #22)

I have squashed the 2 patches and made it destroy any old connection before making a new one. Jacek could you please give it a test?

It's looking good so far; this modified patch version works fine for me while testing both under plain CGI and under apache (prefork) + mod_perl. No apparent side effects like before (not counting that some command line scripts are runnig significantly slower when Zconn caching is not being used any longer). Regarding those scripts: I'm wondering if something like that: my $cache_key = join ('::', (map { $_ // '' } ($server, $async ))); + if ( (!defined($ENV{GATEWAY_INTERFACE})) && defined($context->{"Zconn"}->{$cache_key}) && (0 == $context->{"Zconn"}->{$cache_key}->errcode()) ) { + return $context->{"Zconn"}->{$cache_key}; + } would be good (or at least good enough ;) solution for adressing performance hit on the cmd line scripts? It seems to work OK under CGI and mod_perl, but I'm not sure how reliable checking for GATEWAY_INTERFACE environment variable (especially when being done in the module, and not in the script) may be for other various persistent setups? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 --- Comment #25 from Paul Poulain <paul.poulain@biblibre.com> --- another option, as those scripts are always run through command line : have a local-made zoom connexion, not using C4 connector. Would it be possible ? (if someone think it's Q&D, I agree) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #32334|0 |1 is obsolete| | --- Comment #27 from Chris Cormack <chris@bigballofwax.co.nz> --- Created attachment 32404 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32404&action=edit Bug 12005 : Creating a new zebra connection for each time we need one Zebra is not designed to have persistent connections, under cgi this didn't matter the scripts would get a new connection each time, but under plack we try to use dead connections This patch changes it so plack works the same way that cgi did. To test: Apply this patch Do some searches Check everything still works -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 Jacek Ablewicz <ablewicz@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #32404|0 |1 is obsolete| | --- Comment #28 from Jacek Ablewicz <ablewicz@gmail.com> --- Created attachment 32417 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32417&action=edit Bug 12005 : Creating a new zebra connection for each time we need one Zebra is not designed to have persistent connections, under cgi this didn't matter the scripts would get a new connection each time, but under plack we try to use dead connections This patch changes it so plack works the same way that cgi did. To test: Apply this patch Do some searches Check everything still works Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl> -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12005 Brendan Gallagher <brendan@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #32417|0 |1 is obsolete| | --- Comment #29 from Brendan Gallagher <brendan@bywatersolutions.com> --- Created attachment 32429 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=32429&action=edit Bug 12005 : Creating a new zebra connection for each time we need one Zebra is not designed to have persistent connections, under cgi this didn't matter the scripts would get a new connection each time, but under plack we try to use dead connections This patch changes it so plack works the same way that cgi did. To test: Apply this patch Do some searches Check everything still works Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl> Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com> Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.