[Bug 25292] New: L1 cache too long in Z3950 server (z3950-responder)
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Bug ID: 25292 Summary: L1 cache too long in Z3950 server (z3950-responder) Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: ASSIGNED Severity: normal Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: jonathan.druart@bugs.koha-community.org Reporter: jonathan.druart@bugs.koha-community.org QA Contact: testopia@bugs.koha-community.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |24761 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24761 [Bug 24761] Caching issues in scripts running in daemon mode [OMNIBUS] -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 --- Comment #1 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 103795 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=103795&action=edit Bug 25292: Flushing L1 on every Z3950 server search This patch flushes the L1 cache on each Z3950 server search. There is no need to flush it in init_handler (when the connection is made), the L1 cache is not accessed (this behaviour goes against what I expected). This means each search will fetch values from the L2 cache (ie memcached) and persist the L1 cache (in Perl memory cache) until the result are returned (or longer?). Test plan: % perl misc/z3950_responder.pl --config-dir=/kohadevbox/koha/etc/z3950 % zoomsh ZOOM>connect localhost:2100 ZOOM>set databaseName biblios ZOOM>search x ZOOM>search x Between the 2 searches you can modify the value of SearchEngine, you will notice that the number of hits is different -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com Depends on| |13937 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13937 [Bug 13937] Add an Elasticsearch-compatible Z39.50/SRU daemon -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |frederic@tamil.fr --- Comment #2 from Frédéric Demians <frederic@tamil.fr> --- I can reproduce the test plan on a dual installation (ES/Zebra), and see the result variation depending on selected search engine. So it works. But won't it impact performance? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 --- Comment #3 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Of course there will be a difference, but you will hit L2 (memcached) anyway. It won't be worst than other search (UI). I am not expecting a huge difference here but I have not compared. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |victor@tuxayo.net --- Comment #4 from Victor Grousset/tuxayo <victor@tuxayo.net> --- (In reply to Jonathan Druart from comment #1)
ZOOM>search x ZOOM>search x
Between the 2 searches you can modify the value of SearchEngine, you will notice that the number of hits is different
I don't get what should I do to see a difference in the number of hits. I've searched for few things before the patch. And after the patch (after restarting the responder). Same number of hits. Then I did restart_all. In this session, I switched at some points from ES to Zebra. I'm confused and can't invent a list of steps that leads to something consistently good (after the patch) or bad (before the patch). kohadev-koha@6c8a568fb951:/kohadevbox/koha$ zoomsh ZOOM>connect localhost:2100 ZOOM>set databaseName biblios ZOOM>search shock localhost:2100: 2 hits ZOOM>search the localhost:2100: 192 hits ZOOM>search the localhost:2100: 192 hits ZOOM>search the localhost:2100: 192 hits # I think I switched from ZOOM>search the localhost:2100: 0 hits ZOOM>search the localhost:2100: 0 hits ZOOM>search the localhost:2100: 0 hits ZOOM>search les localhost:2100: 0 hits ZOOM>search yes localhost:2100: 0 hits ZOOM>search shoch localhost:2100: 0 hits ZOOM>search shodck localhost:2100: 0 hits ZOOM>search shock localhost:2100: 0 hits ZOOM>search the localhost:2100: 0 hits second try with more details. kohadev-koha@6c8a568fb951:/kohadevbox/koha$ zoomsh ZOOM>connect localhost:2100 ZOOM>set databaseName biblios ZOOM>search shock localhost:2100: 2 hits ZOOM>search the localhost:2100: 192 hits ZOOM> kohadev-koha@6c8a568fb951:/kohadevbox/koha$ # switch toZebra kohadev-koha@6c8a568fb951:/kohadevbox/koha$ zoomsh ZOOM>connect localhost:2100 ZOOM>set databaseName biblios ZOOM>search shock localhost:2100 error: Temporary system error (Bib-1:2) Cannot connect to upstream server ZOOM>search the localhost:2100 error: Temporary system error (Bib-1:2) Cannot connect to upstream server ZOOM> kohadev-koha@6c8a568fb951:/kohadevbox/koha$ zoomsh ZOOM> kohadev-koha@6c8a568fb951:/kohadevbox/koha$ #restart responder kohadev-koha@6c8a568fb951:/kohadevbox/koha$ zoomsh ZOOM>connect localhost:2100 ZOOM>set databaseName biblios ZOOM>search shock localhost:2100 error: Temporary system error (Bib-1:2) Cannot connect to upstream server ZOOM>search the localhost:2100 error: Temporary system error (Bib-1:2) Cannot connect to upstream server ZOOM> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 --- Comment #5 from Victor Grousset/tuxayo <victor@tuxayo.net> --- The last two logs are with the patch applied. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 --- Comment #6 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- This is all what I do: ZOOM>connect localhost:2100 ZOOM>set databaseName biblios ZOOM>search x localhost:2100: 18 hits # Change the syspref SearchEngine ZOOM>search x localhost:2100: 230 hits This is a ugly way of confirming that the correct value has been retrieve, but you could add a warn statement in the code, in search_handler, to check which value from cache (so easy to test is a syspref) is retrieved. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 --- Comment #7 from Victor Grousset/tuxayo <victor@tuxayo.net> --- After changing the syspref, I must restart ZOOM to have a different number of hits. Is that expected? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 --- Comment #8 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Victor Grousset/tuxayo from comment #7)
After changing the syspref, I must restart ZOOM to have a different number of hits.
Is that expected?
No, please detail what you tried and the result. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 --- Comment #9 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Jonathan Druart from comment #8)
(In reply to Victor Grousset/tuxayo from comment #7)
After changing the syspref, I must restart ZOOM to have a different number of hits.
Is that expected?
No, please detail what you tried and the result.
Hum forget that, you detailed above. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #103795|0 |1 is obsolete| | --- Comment #10 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 107621 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=107621&action=edit Bug 25292: Flushing L1 on every Z3950 server search This patch flushes the L1 cache on each Z3950 server search. There is no need to flush it in init_handler (when the connection is made), the L1 cache is not accessed (this behaviour goes against what I expected). This means each search will fetch values from the L2 cache (ie memcached) and persist the L1 cache (in Perl memory cache) until the result are returned (or longer?). Test plan: % perl misc/z3950_responder.pl --config-dir=/kohadevbox/koha/etc/z3950 % zoomsh ZOOM>connect localhost:2100 ZOOM>set databaseName biblios ZOOM>search x ZOOM>search x Between the 2 searches you can modify the value of SearchEngine, you will notice that the number of hits is different -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 --- Comment #11 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I am really confused, I am pretty sure the original patch was working when I wrote it but... reading the code it couldn't have worked. This new patch should work accordingly to what is advertised in the commit message. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |magnus@libriotech.no --- Comment #12 from Magnus Enger <magnus@libriotech.no> --- Tried to test this on Debian 9.13, but when I try to install Net::Z3950::SimpleServer it says "You have YAZ version 4.2.30; you need 4.2.51 or better. at Makefile.PL line 23", so I'm guessing I would need to have Debian 10. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtj@kohaaloha.com --- Comment #13 from Mason James <mtj@kohaaloha.com> --- (In reply to Magnus Enger from comment #12)
Tried to test this on Debian 9.13, but when I try to install Net::Z3950::SimpleServer it says "You have YAZ version 4.2.30; you need 4.2.51 or better. at Makefile.PL line 23", so I'm guessing I would need to have Debian 10.
hi Magnus, have you tried using the version in the debian.org repo? https://packages.debian.org/stretch/libnet-z3950-simpleserver-perl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #107621|0 |1 is obsolete| | --- Comment #14 from Fridolin Somers <fridolin.somers@biblibre.com> --- Created attachment 114436 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=114436&action=edit Bug 25292: Flushing L1 on every Z3950 server search This patch flushes the L1 cache on each Z3950 server search. There is no need to flush it in init_handler (when the connection is made), the L1 cache is not accessed (this behaviour goes against what I expected). This means each search will fetch values from the L2 cache (ie memcached) and persist the L1 cache (in Perl memory cache) until the result are returned (or longer?). Test plan: % perl misc/z3950_responder.pl --config-dir=/kohadevbox/koha/etc/z3950 % zoomsh ZOOM>connect localhost:2100 ZOOM>set databaseName biblios ZOOM>search x ZOOM>search x Between the 2 searches you can modify the value of SearchEngine, you will notice that the number of hits is different Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m --- Comment #15 from Fridolin Somers <fridolin.somers@biblibre.com> --- Works well when you have both searchengine running. Noticed that daemon fails when you switch to Elasticsearch without having it installed. But that is for another bug. Best regards ;) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #114436|0 |1 is obsolete| | --- Comment #16 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 114976 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=114976&action=edit Bug 25292: Flushing L1 on every Z3950 server search This patch flushes the L1 cache on each Z3950 server search. There is no need to flush it in init_handler (when the connection is made), the L1 cache is not accessed (this behaviour goes against what I expected). This means each search will fetch values from the L2 cache (ie memcached) and persist the L1 cache (in Perl memory cache) until the result are returned (or longer?). Test plan: % perl misc/z3950_responder.pl --config-dir=/kohadevbox/koha/etc/z3950 % zoomsh ZOOM>connect localhost:2100 ZOOM>set databaseName biblios ZOOM>search x ZOOM>search x Between the 2 searches you can modify the value of SearchEngine, you will notice that the number of hits is different Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |21.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 --- Comment #17 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 21.05, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|21.05.00 |21.05.00,20.11.02 released in| | Status|Pushed to master |Pushed to stable --- Comment #18 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 20.11.x for 20.11.02 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Pushed to oldstable CC| |andrew@bywatersolutions.com Version(s)|21.05.00,20.11.02 |21.05.00,20.11.02,20.05.08 released in| | --- Comment #19 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Pushed to 20.05.x for 20.05.08 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|Pushed to oldstable |RESOLVED --- Comment #20 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Not backported to oldoldstable (19.11.x). Feel free to ask if it's needed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25292 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |caroline.cyr-la-rose@inlibr | |o.com -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org