[Bug 6435] New: False "no items" in search results [intranet]
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Bug #: 6435 Summary: False "no items" in search results [intranet] Classification: Unclassified Change sponsored?: --- Product: Koha Version: rel_3_4 Platform: All OS/Version: All Status: NEW Severity: major Priority: P5 Component: Cataloging AssignedTo: gmcharlt@gmail.com ReportedBy: eb@efdss.org QAContact: koha-bugs@lists.koha-community.org CC: ian.walls@bywatersolutions.com This applies only to things catalogued since upgrading to 3.4 The search results for these new records show no items in the hit list. But if you click on the title to bring up the normal view, the items are there, and are fully functional. The OPAC seems unaffected. The records created before we moved to 3.4 are also unaffected. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |chris@bigballofwax.co.nz --- Comment #1 from Chris Cormack <chris@bigballofwax.co.nz> 2011-05-31 12:11:49 UTC --- I cant replicate this, both with and without xslt turned on -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Doug Kingston <dpk@randomnotes.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dpk@randomnotes.org --- Comment #2 from Doug Kingston <dpk@randomnotes.org> 2011-05-31 12:35:49 UTC --- Just to be sure, is it save to rerun remove_items_from_biblioitems.pl? I can rerun to make sure this has been done. (and then rebuild the zebra index) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Elaine Bradtke <eb@efdss.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |eb@efdss.org --- Comment #3 from Elaine Bradtke <eb@efdss.org> 2011-06-13 15:53:22 UTC --- To clarify: Records added before the switch display in the hit list with three columns: Results, Location, and an unlabeled one that contains a list of items for that biblio. The biblios added after the upgrade follow after the ones added before so it's easy to spot the change in our catalogue. The biblios added after the switch all say No items in the third column. But the items are there when you click on the title and even in the hit list, you have a link to Edit items, even when it says no items. The inconsistency is curious. But I cannot find a difference between the records created before and after the switch that would cause this. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Kyle M Hall <kyle.m.hall@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle.m.hall@gmail.com --- Comment #4 from Kyle M Hall <kyle.m.hall@gmail.com> 2011-06-16 11:45:39 UTC --- I just experienced the same problem on a freshly installed test server. All mandatory and optional data was imported at the time of the installation. No settings had been touched as far as I am aware. However, any items that were catalogued were showing in the results as "no items", but the record details listed the items just fine. I re-indexed the catalogue with "misc/migration_tools/rebuild_zebra.pl -b -r -v" and now everything shows up just fine. When I set this installation up, I used the koha zebraqueue daemon to manage the zebraqueue. Could this be the source of the problem? It must be something zebra related for a re-index to fix the problem for me. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #5 from Doug Kingston <dpk@randomnotes.org> 2011-07-06 22:23:01 UTC --- We have now upgraded to 3.4.2 and the problem persists. We can fix it by running './rebuild_zebra.pl -b', but this only fixes it until we update the record and then a search that returns the biblio will again show "No items". This is definitely some sort of Koha to Zebra update issue. The only errors I find recorded in /var/log/koha around that time are: [Wed Jul 06 22:54:29 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:54:29 2011] search.pl: Use of uninitialized value $key in hash element at /usr/share/koha/lib/C4/Templates.pm line 227., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/detail.pl?biblionumb... [Wed Jul 06 22:54:31 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:54:31 2011] search.pl: Use of uninitialized value $key in hash element at /usr/share/koha/lib/C4/Templates.pm line 227., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/search.pl?q=evelyn+s... [Wed Jul 06 22:55:39 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:55:39 2011] addbiblio.pl: Subroutine plugin_parameters redefined at /usr/share/koha/intranet/cgi-bin/cataloguing/value_builder/callnumber.pl line 39., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/search.pl?q=evelyn+s... [Wed Jul 06 22:55:39 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:55:39 2011] addbiblio.pl: Subroutine plugin_javascript redefined at /usr/share/koha/intranet/cgi-bin/cataloguing/value_builder/callnumber.pl line 42., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/search.pl?q=evelyn+s... [Wed Jul 06 22:55:39 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:55:39 2011] addbiblio.pl: Subroutine plugin redefined at /usr/share/koha/intranet/cgi-bin/cataloguing/value_builder/callnumber.pl line 73., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/search.pl?q=evelyn+s... [Wed Jul 06 22:55:59 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:55:59 2011] search.pl: Use of uninitialized value $key in hash element at /usr/share/koha/lib/C4/Templates.pm line 227., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/detail.pl?biblionumb... [Wed Jul 06 22:56:01 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:56:01 2011] search.pl: Use of uninitialized value $key in hash element at /usr/share/koha/lib/C4/Templates.pm line 227., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/search.pl?q=evelyn+s... Zebraqueue daemon is running and appears to be happy. Syslog show the following almost immediately when I update a record: Jul 6 23:12:49 www Koha Zebraqueue [6696]: 1 operations waiting to be run Jul 6 23:12:49 www Koha Zebraqueue [6696]: Executing zebra operations Further ideas are most welcome. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #6 from Doug Kingston <dpk@randomnotes.org> 2011-07-07 11:50:14 UTC --- So it looks like some problem with the zebra incremental update managed by the zebraqueue daemon. Rebuilds of the index (rebuild_zebra.pl -b) resolves the issue temporarily (until the next record update). It appears to be an issue induced by the upgrade we did from v3.2.6 to v3.4.1. We are now on v3.4.2 and the issue persists. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Frère Sébastien Marie <semarie-koha@latrappe.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |semarie-koha@latrappe.fr --- Comment #7 from Frère Sébastien Marie <semarie-koha@latrappe.fr> 2011-07-07 12:57:49 UTC --- I can confirm this issue: we have the same (running 3.4.2), and it was noted after transition from 3.2.x to 3.4.x As workaround I have set cronjobs for launch regulary 'rebuild_zebra.pl -b -a -z'. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA Contact for the bug.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Fridolyn SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolyn.somers@biblibre.co | |m See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=10482 -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #8 from Fridolyn SOMERS <fridolyn.somers@biblibre.com> --- I think this bug is a consequence of removing items form MARC. When using rebuild_zebra.pl, in get_raw_marc_record() : - without "--noxml" option the items are added to MARC::Record in GetMarcBiblio() call. - with "--noxml" option, MARC::Record is build from biblioitems.marc and items are not added. This might be corrected by Bug 10482. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #9 from Doug Kingston <dpk@randomnotes.org> --- This is an interesting observation Fridolyn. We are running 3.10.6. BIB_INDEX_MODE=dom. We have recently started seeing problems again with item records disappearing when the biblio record is updated. If you do a search, you will get back the matching biblios, and the modified one will show "No items.", but if you drill into the result, the item *is* displayed, so its in the Koha system. This happens with biblios that have a single item record. If I reindex with "rebuild_zebra.pl -b -v -x" then the item reappears in catalog searchs. Could it be that after the biblio change, an incomplete MARC record is exported for update? We are using the koha-index-daemon. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #10 from Fridolyn SOMERS <fridolyn.somers@biblibre.com> --- You mean zebraqueue-daemon.pl ? -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #11 from Doug Kingston <dpk@randomnotes.org> --- I was using koha-index-daemon (http://search.cpan.org/~fredericd/Koha-Contrib-Tamil-0.029/bin/koha-index-da...) but clearly this does not correctly export all records. So I have stopped and reverted back to the less elegant (but functioning) koha-zebraqueue-daemon (zebraqueue-daemon.pl). We should either get this other daemon fixed or removed. I forget how I got pointed to it originally. Our problem is now resolved. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #12 from Jared Camins-Esakov <jcamins@cpbibliography.com> --- zebraqueue_daemon is broken and was removed from Koha several versions ago. Do not use it. Your best bet would be to either use rebuild_zebra and teach your staff to deal with the delay, or to find someone who can troubleshoot the koha-index-daemon from Tamil. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |frederic@tamil.fr -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #13 from Doug Kingston <dpk@randomnotes.org> --- OK, I have gone to the cron job. I will look into an alternative wrapper daemon that just checks the mysql queue and calls rebuild only when necessary. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #14 from Doug Kingston <dpk@randomnotes.org> --- Created attachment 19267 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=19267&action=edit Patch to add daemon mode to rebuild_zebra.pl This diff adds a daemon mode to rebuild_zebra.pl. By polling the queue every few seconds from a running process, we can achieve very fast indexing response with low system overhead. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #15 from Frédéric Demians <frederic@tamil.fr> --- Your patch contains: sleep 5; This must be configurable. 5 seconds is too small for large catalogs. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Doug Kingston <dpk@randomnotes.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #19267|0 |1 is obsolete| | --- Comment #16 from Doug Kingston <dpk@randomnotes.org> --- Created attachment 19710 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=19710&action=edit Improved patch to add daemon mode to rebuild_zebra.pl This improved patch adds a variable to set the sleep time between checks for new records to index. The check is very cheap and not dependent on the size of the catalog so we make it every 5 seconds. Any records found (probably just one), are indexed by zebra incrementally. This will make new records nearly instantly available for search. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #17 from Fridolyn SOMERS <fridolyn.somers@biblibre.com> --- (In reply to Doug Kingston from comment #16)
Created attachment 19710 [details] [review] Your patch is very odd. Did you create it with git format-patch ?
-- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Liz Rea <liz@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply CC| |liz@catalyst.net.nz --- Comment #18 from Liz Rea <liz@catalyst.net.nz> --- Hi Doug, I would like to test this patch, however it appears to be formatted incorrectly - please see http://wiki.koha-community.org/wiki/Version_Control_Using_Git for help on setting up git and sending in patches. If you are having trouble with any of that, do say and I'll see about making a git formatted patch for you - but please have a try at doing it with git first. :) Thanks, Liz -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtompset@hotmail.com --- Comment #19 from M. Tompsett <mtompset@hotmail.com> --- Created attachment 20574 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=20574&action=edit My attempt to generate a useful patch from Doug's bizarre diff file. I wanted to accomplish something today. :) The one problem with his patch that I noticed thus far would include: -- Failed to update the HELP to give information regarding the parameters added. test: $ ./misc/migration_tools/rebuild_zebra.pl --help Take this patch, and fix it up, Doug. :) -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|False "no items" in search |Added daemon mode |results [intranet] |parameters to | |rebuild_zebra.pl -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #20574|0 |1 is obsolete| | --- Comment #20 from M. Tompsett <mtompset@hotmail.com> --- Created attachment 20575 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=20575&action=edit Convert Doug's diff into a patch. Oops! Had a space problem that would have triggered a QA issue. :) -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Doug Kingston <dpk@randomnotes.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #19710|0 |1 is obsolete| | Attachment #20575|0 |1 is obsolete| | --- Comment #21 from Doug Kingston <dpk@randomnotes.org> --- Created attachment 21097 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21097&action=edit Proper GIT patch for the daemon mode addition to rebuild_zebra.pl This patch is GIT generated against HEAD and includes updates to the help as requested. Once this patch has been vetted and tested by others, we should probably update the guidance in the various INSTALL files to move away from periodic cron jobs. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #22 from M. Tompsett <mtompset@hotmail.com> --- "Seconcds to sleep between checks" typo in the patch. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Doug Kingston <dpk@randomnotes.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #21097|0 |1 is obsolete| | --- Comment #23 from Doug Kingston <dpk@randomnotes.org> --- Created attachment 21099 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21099&action=edit Proper GIT patch for the daemon mode addition to rebuild_zebra.pl (w/typo fixed) Trivial update to prior patch. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff CC|mtompset@hotmail.com | --- Comment #24 from M. Tompsett <mtompset@hotmail.com> --- Changed status, because Doug forgot. :) -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 I'm just a bot <gitbot@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply CC| |gitbot@bugs.koha-community. | |org When did the bot| |2013-09-24 last check this| | --- Comment #25 from I'm just a bot <gitbot@bugs.koha-community.org> --- Applying: Bug 6435 Add daemon mode to rebuild_zebra.pl (w/typo fix) Repository lacks necessary blobs to fall back on 3-way merge. Cannot fall back to three-way merge. Patch failed at 0001 Bug 6435 Add daemon mode to rebuild_zebra.pl (w/typo fix) The copy of the patch that failed is found in: /home/christopher/git/koha/.git/rebase-apply/patch When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort". -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #26 from Chris Cormack <chris@bigballofwax.co.nz> --- Ahh I see the problem, the first patch has been obsoleted, there should be 2 patches on this bug, will restore the first one and test again. Yay for the bot catching this! -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #21097|1 |0 is obsolete| | -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Signed Off -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #21097|0 |1 is obsolete| | --- Comment #27 from Chris Cormack <chris@bigballofwax.co.nz> --- Created attachment 21390 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21390&action=edit Bug 6435 Add daemon mode to rebuild_zebra.pl This change adds code to check the zebraqueue table with a cheap SQL query and a daemon loop that checks for new entries and processes them incremantally before sleeping for a controllable number of seconds. The default is 5 seconds which provides a near realtime search index update. This is desirable particularly for libraries that are doing active catalogue updating. Help text updated. Tested against a live 3.12 system. Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz> Does what it says on the box, I wonder if it leaks ram at all, would have to run for a long time. But I'm happy to sign off as it doesn't break the existing functionality, while adding new. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #21099|0 |1 is obsolete| | --- Comment #28 from Chris Cormack <chris@bigballofwax.co.nz> --- Created attachment 21391 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=21391&action=edit Bug 6435 Add daemon mode to rebuild_zebra.pl (w/typo fix) This change adds code to check the zebraqueue table with a cheap SQL query and a daemon loop that checks for new entries and processes them incremantally before sleeping for a controllable number of seconds. The default is 5 seconds which provides a near realtime search index update. This is desirable particularly for libraries that are doing active catalogue updating. Help text updated. Tested against a live 3.12 system. Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz> -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #29 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Please provide a step by step test plan to make this easier to QA. Thx! -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |dpk@randomnotes.org -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #30 from Doug Kingston <dpk@randomnotes.org> --- Apply both patches above. One is the real code and the small one fixes a typo. Suggested test plan: Invoke the daemon exercising all the options I added and observe it doing the intended updates in line with the update interval. 1. Set up a test koha configuration (populate with data if you wish) 2. Ensure that the zebra database is up to date (rebuild_zebra.pl -b -a) 3. Stop any cronjob that may be updating zebra with rebuild_zebra.pl 4. Start rebuild_zebra in daemon mode (rebuild_zebra.pl -daemon -sleep 3 -b -a v 5. TEST: observe the periodic message "queued records: 0" every 3 seconds. this will test the use of -sleep option. 6. Add an authority. 7. TEST: observe the daemon pick up the authority change and update zebra (which should be within 3 seconds of completing the add operation) 8. TEST: check that you can search for the new authority from the admin interface 9. Add a biblio. 10. TEST: observe teh daemon pickup up the biblio change and update zebra (which should be within 3 seconds of completing the add operation) 11. TEST: check that you can search for the new biblio in either OPAC or admin interface. Possibly repeat steps 6-11 with authority and biblio modification and deletion. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Doug Kingston <dpk@randomnotes.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|dpk@randomnotes.org |katrin.fischer@bsz-bw.de -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|katrin.fischer@bsz-bw.de |dpk@randomnotes.org --- Comment #31 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Doug, you wrote the patch, so you are assignee :) -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|3.4 |master -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Doug Kingston <dpk@randomnotes.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Added daemon mode |[ENH] Added daemon mode |parameters to |parameters to |rebuild_zebra.pl |rebuild_zebra.pl -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #32 from Doug Kingston <dpk@randomnotes.org> --- Created attachment 22039 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22039&action=edit Add locking to rebuild_zebra to prevent races This patch adds flock based locking for rebuild_zebra.pl on a per-instance basis. This prevents races between full rebuilds and background incremental updates from the zebraqueue table in the database. The race condition exists whether you are doing incremental updates with a periodic cronjob or with the new daemon mode. Suppose you start a full rebuild at time T0 which will take until T20 to extract the records. Suppose also at T10, a biblio or auth is updated and processed through the zebraqueue by T15. In this situation the updated record in zebra will be overwritten when the full rebuild records are uploaded to zebra after T20. We prevent this by only allowing one rebuild_zebra per koha instance to be running at one time. When running in daemon mode, incremental updates will be skipped while a full rebuild is running, and resume afterwards. A full rebuild or other adhoc request will wait for any previous lock to clear. Tested by flocking the lock file while invoking rebuild_zebra.pl in various modes (daemon, adhoc zebraqueue task, and full rebuild) using flock program I will attach to bug. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #33 from Doug Kingston <dpk@randomnotes.org> --- Created attachment 22040 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22040&action=edit Small perl program to flock files and run commands This program can be used to test the flock functionality introduced in patch 0003 also attached to this bug. Test I performed that QA can replicate (modify for your test instance name): flock the file, and make sure a full rebuild waits for the lock to clear before proceeding window 1: flock /var/lock/koha_rebuild_zebra_koha_test/lock sleep 15 window 2: ./rebuild_zebra.pl -b -a -v window 2 should wait for window 1's sleep to time out before proceeding. test that an flock from another process with cause the daemon mode to pause updates and resume when the flock is removed window 1: ./rebuild_zebra.pl -daemon -sleep 2 -v -z -b -a window 2: flock /var/lock/koha_rebuild_zebra_koha_test/lock sleep 15 start window 2 after window 1 is printing a message every 2 seconds. when you launch the sleep, it will stop the updates which will resume when the sleep times out. remove /var/log/koha_rebuild_zebra_koha_test and make sure its recreated on the next invocation of rebuild_zebra.pl. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com --- Comment #34 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Any chance we can move the 'Add locking to rebuild_zebra to prevent races' followup patch to a dependant bug instead of adding it here.. That way QA can go on with the deamon mode patch without being held in waiting for a further signoff. (PS, if you do that, I'll test and sign off on it). -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Doug Kingston <dpk@randomnotes.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |11078 -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #35 from Doug Kingston <dpk@randomnotes.org> --- I have created separate bug 11078 to track the race condition and its locking fix. 11078 depends on this bug. 6435 does not create any race condition that did not already exist. The patch for locking and the flock test program are attached to 11078. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Doug Kingston <dpk@randomnotes.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #22039|0 |1 is obsolete| | Attachment #22040|0 |1 is obsolete| | --- Comment #36 from Doug Kingston <dpk@randomnotes.org> --- Created attachment 22080 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22080&action=edit empty file to allow obsoleting other files. Deleting unneeded files which have moved to a new bug 11078. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #22080|0 |1 is obsolete| | --- Comment #37 from M. Tompsett <mtompset@hotmail.com> --- Comment on attachment 22080 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22080 empty file to allow obsoleting other files. There is no need for this file. To obsolete files: Click on "Details" beside the attachment name. Click on "Edit Details" beside the name at the top. Check the obsolete box near the middle/bottom right. Click the appropriate save/submit button. To un-obsolete files: Click on link to show all the obsoleted items Click on "Details" beside the attachment name. Click on "Edit Details" beside the name at the top. Un-check the obsolete box near the middle/bottom right. Click the appropriate save/submit button. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #38 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 22201 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22201&action=edit Bug 6345 Add locking to rebuild_zebra This patch adds flock based locking for rebuild_zebra.pl on a per-instance basis. This prevents races between full rebuilds and background incremental updates from the zebraqueue table in the database. The race condition exists whether you are doing incremental updates with a periodic cronjob or with the new daemon mode. Suppose you start a full rebuild at time T0 which will take until T20 to extract the records. Suppose also at T10, a biblio or auth is updated and processed through the zebraqueue by T15. In this situation the updated record in zebra will be overwritten when the full rebuild records are uploaded to zebra after T20. We prevent this by only allowing one rebuild_zebra per koha instance to be running at one time. When running in daemon mode, incremental updates will be skipped while a full rebuild is running, and resume afterwards. A full rebuild or other adhoc request will wait for any previous lock to clear. Tested by flocking the lock file while invoking rebuild_zebra.pl in various modes (daemon, adhoc zebraqueue task, and full rebuild) using flock program I will attach to bug. http://bugs.koha-community.org/show_bug.cgi?id=11078 Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #22201|0 |1 is obsolete| | --- Comment #39 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Comment on attachment 22201 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22201 Bug 6345 Add locking to rebuild_zebra Oops.. wrong bug. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #40 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I think this doesn't work quite right if you use the daemon mode only for -b not for -a. The command I ran was: ./misc/migration_tools/rebuild_zebra.pl -daemon -sleep 4 -b -v -z Then I added a new authority, it keeps repeating the following lines: queued records: 1 skipping authorities ==================== exporting biblio ==================== Records exported: 0 Records exported: 0 ==================== REINDEXING zebra ==================== The record is not reindexed, but Koha keeps trying. I think this is because the SQL statement doesn't differentiate between authority and biblio records. Doug, could you take a look? -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Doug Kingston <dpk@randomnotes.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #21390|0 |1 is obsolete| | Attachment #21391|0 |1 is obsolete| | --- Comment #41 from Doug Kingston <dpk@randomnotes.org> --- Created attachment 22457 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22457&action=edit [ENH] Bug 6435 Add daemon mode to rebuild_zebra.pl (revised 10/27) (combines previous patches and addresses recent QA issue) This change adds code to check the zebraqueue table with a cheap SQL query and a daemon loop that checks for new entries and processes them incremantally before sleeping for a controllable number of seconds. The default is 5 seconds which provides a near realtime search index update. This is desirable particularly for libraries that are doing active catalogue updating. The query is adjusted based on whether -a, -b, or -a -b are specified. Help text updated. Tested against a live 3.12 system. Note that this fix will benefit from the fix to lack of locking (bug 11078) -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 M. Tompsett <mtompset@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff CC| |mtompset@hotmail.com --- Comment #42 from M. Tompsett <mtompset@hotmail.com> --- I'm just curious if you would want to run daemon mode for only biblios or only authorities. If this process is going to run in daemon mode, would you not want both to be picked up? Otherwise, you'd have to run two daemons (one for bibs and one for auths). Changing the status to needs sign off, since this question doesn't really affect whether the code does what it is supposed to do, but I haven't tested it. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #43 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Yes, you'd probably want both, but for example if your library don't uses authorities and you don't turn it on and then someone catalogs an authority for testing... it shouldn't lead to trouble. It's just something that could go wrong that we might want to avoid either by documentation or by catching it in the code. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #44 from Doug Kingston <dpk@randomnotes.org> --- I addressed the recent QA issue on 10/28 with the updated patch. Can this be re-reviewed? -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #22457|0 |1 is obsolete| | --- Comment #45 from Chris Cormack <chris@bigballofwax.co.nz> --- Created attachment 22851 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22851&action=edit Bug 6435 Add daemon mode to rebuild_zebra.pl This change adds code to check the zebraqueue table with a cheap SQL query and a daemon loop that checks for new entries and processes them incremantally before sleeping for a controllable number of seconds. The default is 5 seconds which provides a near realtime search index update. This is desirable particularly for libraries that are doing active catalogue updating. The query is adjusted based on whether -a, -b, or -a -b are specified. Help text updated. Tested against a live 3.12 system. Note that this fix will benefit from the fix to lack of locking (bug 11078) Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #22851|0 |1 is obsolete| | --- Comment #46 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 22859 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22859&action=edit Bug 6435 Add daemon mode to rebuild_zebra.pl This change adds code to check the zebraqueue table with a cheap SQL query and a daemon loop that checks for new entries and processes them incremantally before sleeping for a controllable number of seconds. The default is 5 seconds which provides a near realtime search index update. This is desirable particularly for libraries that are doing active catalogue updating. The query is adjusted based on whether -a, -b, or -a -b are specified. Help text updated. Tested against a live 3.12 system. Note that this fix will benefit from the fix to lack of locking (bug 11078) Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Signed Off --- Comment #47 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- I'm up and down on passing this one.. discussed it with Katrin.. 1. Passes QA script (after I fixed some minor white space errors - Doug, please take a look at http://blog.bigballofwax.co.nz/2011/12/15/fixing-whitespace-when-apply-patch... to stop these in their tracks before they hit QA with future patches ;) ) 2. Tested with various combinations of options: -a -daemon, -b -daemon, -a -b -daemon, -a -b -x -daemon, all worked as expected along with not breaking existing options. One niggle worth discussing: The intention of this patch is clear, to allow regular rebuilds as a daemon.. However, the patch allows all/any params to be passed along with the daemon mode. This slightly scares me as it becomes rather easy to break the system using the wrong combination of options. (for instance, running with -r would mean that every time a catalogue update happens, the zebra index disappears moments later whilst a full rebuild is happening) I don't think it's a QA fail, but it may be worth documenting that certain options don't work well with a daemon model. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #48 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Follow-up note 1: I also tested the -sleep option and it worked well. Follow-up note 2: As an afterthought, should the -z be implied by daemon mode? -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugs.koha-community. | |org/bugzilla3/show_bug.cgi? | |id=11078 -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |gmcharlt@gmail.com --- Comment #49 from Galen Charlton <gmcharlt@gmail.com> --- (In reply to Martin Renvoize from comment #48)
Follow-up note 1: I also tested the -sleep option and it worked well. Follow-up note 2: As an afterthought, should the -z be implied by daemon mode?
In response to this and the previous comment, my view is that --daemon should imply (and be) "here's how you keep your Zebra index up to date." IOW, I think --daemon should imply the following options: -z -a -b It should exclude (by refusing to start or by simply ignoring) the following options: -r -s -k -h -y -where -length -offset -I | -skip-index I wouldn't consider this a blocker for putting the patch in master, but lean towards considering a it blocker for putting in a stable branch. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #50 from Doug Kingston <dpk@randomnotes.org> --- I can try to pull that patch together this weekend. -Doug- -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #51 from Galen Charlton <gmcharlt@gmail.com> --- (In reply to Doug Kingston from comment #50)
I can try to pull that patch together this weekend.
Great, thanks! -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #52 from Doug Kingston <dpk@randomnotes.org> --- Created attachment 22986 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=22986&action=edit QA Patch - Make daemon mode imply -z -a -b and abort on incompatible flags -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master --- Comment #53 from Galen Charlton <gmcharlt@gmail.com> --- Pushed to master. Thanks, Doug! Also, this appears to be your first patch accepted into Koha. Congratulation! -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 --- Comment #54 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice work Doug, excellent follow through. -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 Fridolin SOMERS <fridolyn.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable --- Comment #55 from Fridolin SOMERS <fridolyn.somers@biblibre.com> --- This patch has been pushed to 3.14.x, will be in 3.14.1 Contains 3 patches : 08c8bea Bug 6435: (follow-up) make -daemon really imply -a and -b ae9e02a Bug 6435: (follow-up) rebuild_zebra -daemon option now smarter e7482ec Bug 6435: Add daemon mode to rebuild_zebra.pl -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6435 M. de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|11078 | -- You are receiving this mail because: You are the QA Contact for the bug. You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org