[Bug 7286] New: fix rebuild_zebra.pl to add rebuild_zebra_sliced.zsh and
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Bug #: 7286 Summary: fix rebuild_zebra.pl to add rebuild_zebra_sliced.zsh and Classification: Unclassified Change sponsored?: --- Product: Koha Version: rel_3_6 Platform: All OS/Version: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching AssignedTo: gmcharlt@gmail.com ReportedBy: stephane.delaune@biblibre.com QAContact: ian.walls@bywatersolutions.com -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|gmcharlt@gmail.com |stephane.delaune@biblibre.c | |om -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|fix rebuild_zebra.pl to add |fix rebuild_zebra.pl to add |rebuild_zebra_sliced.zsh |rebuild_zebra_sliced.zsh |and | -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 --- Comment #1 from stephane.delaune@biblibre.com 2011-12-02 17:20:40 UTC --- Created attachment 6544 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6544 add rebuild_zebra_sliced.zsh (to rebuild zebra by batch automatically) add rebuild_zebra_sliced.zsh (to rebuild zebra by batch automatically) and its support in rebuild-zebra.pl -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - low |PATCH-Sent Patch Status|--- |Needs Signoff -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 --- Comment #2 from stephane.delaune@biblibre.com 2011-12-02 17:21:57 UTC --- just run misc/migration_tools/rebuild_zebra_sliced.zsh to test it -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #3 from Katrin Fischer <katrin.fischer@bsz-bw.de> 2011-12-21 07:23:43 UTC --- Hi Stephane, can you add a bit more information about how this works and what it is doing/changing? -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com --- Comment #4 from Tomás Cohen Arazi <tomascohen@gmail.com> 2012-01-03 15:14:28 UTC --- It looks like a wrapper to the rebuild_zebra.pl script to filter the records being indexed by 'where' clauses or caount limits. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |ASSIGNED CC| |paul.poulain@biblibre.com Depends on| |7246 --- Comment #5 from Paul Poulain <paul.poulain@biblibre.com> 2012-02-03 09:16:00 UTC --- Stéphane, rebuild_zebra_sliced relies on rebuild_zebra length and min options. Look at bug 7246, as i've been suggested to change the name or the variable, length/min were not good terms. + on biblibre/master min and length where/are inverted, i've reordered them correctly. rebuild_zebra_sliced must be updated (not a big deal) Removing "need signoff", as it can't be signed -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 --- Comment #6 from stephane.delaune@biblibre.com 2012-02-06 11:22:30 UTC --- Created attachment 7463 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7463 the 7286 now depends of 7246 so it's the first of 2 patches the 7286 now depends of 7246 so it's the first of 2 patches -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #6544|0 |1 is obsolete| | --- Comment #7 from stephane.delaune@biblibre.com 2012-02-06 11:23:41 UTC --- Created attachment 7464 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7464 the second of 2 patches (its the 7286) the second of 2 patches (its the 7286) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 --- Comment #8 from stephane.delaune@biblibre.com 2012-02-06 11:25:12 UTC --- Created attachment 7465 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7465 all patches (7246 and 7286) into one file all patches (7246 and 7286) into one file -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #7463|0 |1 is obsolete| | Attachment #7464|0 |1 is obsolete| | Attachment #7465|0 |1 is obsolete| | --- Comment #9 from stephane.delaune@biblibre.com 2012-02-06 13:06:44 UTC --- Created attachment 7466 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7466 proposed patch the updated patch to merge after 7246 -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 --- Comment #10 from stephane.delaune@biblibre.com 2012-02-06 13:16:05 UTC --- I proposed an updated patch (to merge after 7246). this script rebuild the zebra index recursively. he take 3 arguments : first specify how many records we must try at once second is the record number to start third is the record number to end the argumentys are optionnal (default value for the first is 10000, for the second is 0, and for the third is the las biblionumber in db) EXAMPLE to run the script on 800 000 biblios : ./reindex.zsh 50000 0 800000 will reindex the DB from biblionumber 0 to 800000, with chunks of 50k biblios or to reindexe all biblio step by step (10000 by 10000, it's the default values) : ./reindex.zsh -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 --- Comment #11 from stephane.delaune@biblibre.com 2012-02-06 13:25:20 UTC --- in the last comment the script's name is wrong in examples : EXAMPLE to run the script on 800 000 biblios : ./misc/migration_tools/rebuild_zebra_sliced.zsh 50000 0 800000 will reindex the DB from biblionumber 0 to 800000, with chunks of 50k biblios or to reindexe all biblio step by step (10000 by 10000, it's the default values) : ./misc/migration_tools/rebuild_zebra_sliced.zsh the script can be launch from from any path (after exporting koha environment variables) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 --- Comment #12 from stephane.delaune@biblibre.com 2012-02-10 11:14:53 UTC --- Created attachment 7562 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7562 followup adding an option in rebuild_zebra_sliced.zsh Adds an optionnal fourth option called "TEMPDIRECTORY" to rebuild_zebra_sliced.zsh to define a temporary directory to rebuild_zebra (default value : /tmp/rebuild) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|PATCH-Sent (DO NOT USE) |P3 Status|Needs Signoff |Patch doesn't apply -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #7466|0 |1 is obsolete| | Attachment #7562|0 |1 is obsolete| | --- Comment #13 from stephane.delaune@biblibre.com 2012-02-13 16:04:06 UTC --- Created attachment 7629 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7629 proposed patch (to merge after 7246) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|fix rebuild_zebra.pl to add |Introduce |rebuild_zebra_sliced.zsh |rebuild_zebra_sliced.zsh to | |recursively rebuild zebra & | |be error proof -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 stephane.delaune@biblibre.com changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P3 |P2 -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #7629|0 |1 is obsolete| | --- Comment #14 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 10095 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=10095&action=edit Bug 7286 [Signed] Rebuild Zebra Sliced Works as expected, very useful for finding errant biblios that don't want to index. Minor comment, when running from command line some further feedback might be nice as currently if it's a slow index with large chunk size the screen is left blank for a long time leaving the user unsure whether anything is happening. Not enough to hold back sign-off. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off CC| |martin.renvoize@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dpavlin@rot13.org --- Comment #15 from Dobrica Pavlinusic <dpavlin@rot13.org> --- Does this script really need zsh? While I don't have anything against zsh as interactive shell, I think it's bad practice to introduce additional dependency to zsh for every Koha installation which want to use this script. I for one don't use zsh and don't have it installed on servers. Than again, this might be my Debian bias -- system scripts should use /bin/sh and not /bin/bash /bin/zfs, /bin/t?csh.... -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 --- Comment #16 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 10598 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=10598&action=edit Bug 7286: rebuild_zebra_sliced for biblios and authorities Complete rewrite of rebuild_zebra_sliced.zsh (renamed to .sh). Main improvements are: - both biblio and authority records are handled - records are exported only once It also add an option --skip-index to rebuild_zebra.pl that permit to use rebuild_zebra.pl as an 'export only' script. As the original patch is already signed-off, I do not obsolete it so that it could be pushed in master, but I think this new version could easily replace the old one. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #10598|0 |1 is obsolete| | --- Comment #17 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 10621 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=10621&action=edit Bug 7286: rebuild_zebra_sliced for biblios and authorities Fix some bugs: - add 'zebraidx commit' after each sucessful call to 'zebraidx update' - option -r is taken into account -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #10095|0 |1 is obsolete| | Attachment #10621|0 |1 is obsolete| | --- Comment #18 from Mason James <mtj@kohaaloha.com> --- Created attachment 10655 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=10655&action=edit Bug 7286: rebuild_zebra_sliced for biblios and authorities Complete rewrite of rebuild_zebra_sliced.zsh (renamed to .sh). Main improvements are: - both biblio and authority records are handled - records are exported only once It also add an option --skip-index to rebuild_zebra.pl that permit to use rebuild_zebra.pl as an 'export only' script. Description: Index Koha records by chunks. It is useful when some record causes errors and stop the indexation process. With this script, if indexation of one chunk fails, chunk is splitted in 2 (or 3) chunks, and indexation continue on these chunks. rebuild_zebra.pl is called only once to export records. Splitting and indexing is handled by this script (using yaz-marcdump and zebraidx). Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA CC| |mtj@kohaaloha.com Version|rel_3_6 |master QA Contact|koha.sekjal@gmail.com |mtj@kohaaloha.com --- Comment #19 from Mason James <mtj@kohaaloha.com> --- Passing QA on this... looks good and works well for me -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master Version|master |rel_3_8 --- Comment #20 from Paul Poulain <paul.poulain@biblibre.com> --- Patch pushed, It's safe and I think it's highly valuable to backport it to 3.8, as it's really a wonderful tool to index large DBs with a few invalid records ! It's a new script, so won't break anything -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |chris@bigballofwax.co.nz --- Comment #21 from Chris Cormack <chris@bigballofwax.co.nz> --- This patch changes rebuild_zebra.pl as well, so not just a new script. I will do some testing to make sure that still works fine (it looks like it should) And will push to 3.8.x if it is ok. -- You are receiving this mail because: You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7286 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable --- Comment #22 from Chris Cormack <chris@bigballofwax.co.nz> --- If you don't pass the new flag to rebuild_zebra.pl nothing changes, so it's a safe change. Pushed to 3.8.x, will be in 3.8.3 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org