[Bug 7246] New: rebuild_zebra.pl --limit option to allow partial re-indexing
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Bug #: 7246 Summary: rebuild_zebra.pl --limit option to allow partial re-indexing Classification: Unclassified Change sponsored?: --- Product: Koha Version: unspecified Platform: All OS/Version: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching AssignedTo: gmcharlt@gmail.com ReportedBy: dpavlin@rot13.org QAContact: ian.walls@bywatersolutions.com Partial re-indexing is mostly useful when experimenting with Zebra configuration and your test database is too large to be re-indexed for each trial. I'm submitting it after I added this twice to source code of rebuild_zebra.pl and I hope it might prove useful to others also. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 --- Comment #1 from Dobrica Pavlinusic <dpavlin@rot13.org> 2011-11-19 11:02:31 UTC --- Created attachment 6345 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6345 rebuild_zebra.pl --limit number for partial reindexing Partial re-indexing is mostly useful when experimenting with Zebra configuration and your test database is too large to be re-indexed for each trial. I'm submitting it after I added this twice to source code of rebuild_zebra.pl and I hope it might prove useful to others also. Test scenario: 1. pick koha instance for which you don't mind truncating results in search 2. re-run rebuild_zebra.pl and add --limit 1000 -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #6345|0 |1 is obsolete| | --- Comment #2 from Dobrica Pavlinusic <dpavlin@rot13.org> 2011-11-19 12:01:40 UTC --- Created attachment 6347 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6347 Bug 7246 - rebuild_zebra.pl --limit number for partial reindexing Partial re-indexing is mostly useful when experimenting with Zebra configuration and your test database is too large to be re-indexed for each trial. I'm submitting it after I added this twice to source code of rebuild_zebra.pl and I hope it might prove useful to others also. Test scenario: 1. pick koha instance for which you don't mind truncating results in search 2. re-run rebuild_zebra.pl and add --limit 1000 -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Change sponsored?|--- |Sponsored -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|gmcharlt@gmail.com |dpavlin@rot13.org -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |frederic@tamil.fr --- Comment #3 from Frédéric Demians <frederic@tamil.fr> 2011-11-20 11:25:47 UTC --- Great idea. I'd personally prefer two options: --offset and --count, or, better, any mean to select an interval of biblionumbers, or specific biblionumbers... -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 --- Comment #4 from Dobrica Pavlinusic <dpavlin@rot13.org> 2011-11-20 18:20:34 UTC --- I was thinking about something like --where 'biblionumber > 100 and biblionumber < 200' which would allow precise selection of interesting range. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - low |PATCH-Sent Patch Status|--- |Needs Signoff -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|unspecified |master -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 --- Comment #5 from Frédéric Demians <frederic@tamil.fr> 2011-11-21 06:43:16 UTC --- (In reply to comment #4)
I was thinking about something like --where 'biblionumber > 100 and biblionumber < 200' which would allow precise selection of interesting range.
Or even, a --select option which would allow for example: --select "SELECT biblionumber FROM biblio WHERE frameworkcode='MONO'" --select "SELECT biblionumber FROM biblio WHERE datecreated > '2010' --select "SELECT biblionumber FROM biblioitems WHERE itemtype='MONO'" -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 --- Comment #6 from Liz Rea <wizzyrea@gmail.com> 2011-12-31 22:28:39 UTC --- Created attachment 6994 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6994 [Signed Off] Bug 7246 - rebuild_zebra.pl --limit number for partial reindexing Partial re-indexing is mostly useful when experimenting with Zebra configuration and your test database is too large to be re-indexed for each trial. I'm submitting it after I added this twice to source code of rebuild_zebra.pl and I hope it might prove useful to others also. Test scenario: 1. pick koha instance for which you don't mind truncating results in search 2. re-run rebuild_zebra.pl and add --limit 1000 http://bugs.koha-community.org/show_bug.cgi?id=7246 Signed-off-by: Liz Rea <wizzyrea@gmail.com> Verified that --limit 3 only exports 3 records of the indexing batch. I think this is a useful option for testers and would like to see it in. The other suggestions can be implemented at a later time. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Liz Rea <wizzyrea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #6347|0 |1 is obsolete| | Attachment #6994|0 |1 is obsolete| | --- Comment #7 from Liz Rea <wizzyrea@gmail.com> 2011-12-31 22:30:28 UTC --- Created attachment 6995 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6995 [Signed Off] Bug 7246 - rebuild_zebra.pl --limit number for partial reindexing Partial re-indexing is mostly useful when experimenting with Zebra configuration and your test database is too large to be re-indexed for each trial. I'm submitting it after I added this twice to source code of rebuild_zebra.pl and I hope it might prove useful to others also. Test scenario: 1. pick koha instance for which you don't mind truncating results in search 2. re-run rebuild_zebra.pl and add --limit 1000 http://bugs.koha-community.org/show_bug.cgi?id=7246 Signed-off-by: Liz Rea <wizzyrea@gmail.com> Verified that --limit 3 only exports 3 records of the indexing batch. I think this is a useful option for testers and would like to see it in. The other suggestions can be implemented at a later time. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Liz Rea <wizzyrea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wizzyrea@gmail.com Patch Status|Needs Signoff |Signed Off -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 --- Comment #8 from Paul Poulain <paul.poulain@biblibre.com> 2012-01-17 16:15:21 UTC --- Created attachment 7200 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7200 Bug 7246 add min/ofset and WHERE options to rebuild_zebra This patch reimplement a feature that is on biblibre/master for Koha-community/master It adds 4 parameters: * ofset = the ofset of record. Say 1000 to start rebuilding at the 1000th record of your database * min = how many records to export. Say 400 to export only 400 records * WHERE = add a where clause to rebuild only a given itemtype, or anything you want to filter on * l = how many items should be export with biblios. This is a usefull option when you have records with so many items that they can result in a record higher than 99999bytes, that zebra don't like at all Another improvement resulting from ofset & min limit is the rebuild_zebra_sliced.zsh that will be submitted in another patch. _sliced will slice your all database in small chunks, and, if something went wrong for a given slice, will slice the slice, and repeat, until you reach a slice size of 1, showing which record is wrong in your database. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #6995|0 |1 is obsolete| | --- Comment #9 from Paul Poulain <paul.poulain@biblibre.com> 2012-01-17 16:16:49 UTC --- Comment on attachment 6995 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=6995 [Signed Off] Bug 7246 - rebuild_zebra.pl --limit number for partial reindexing I've submitted another patch, that does more than your (it implement options suggested by Frederic for example). Obsoleting this one and switching back to needs signoff, liz or anyone else, thx for your feedback ! -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Needs Signoff CC| |paul.poulain@biblibre.com -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #7200|0 |1 is obsolete| | --- Comment #10 from Jared Camins-Esakov <jcamins@cpbibliography.com> 2012-01-29 13:50:57 UTC --- Created attachment 7379 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7379 Bug 7246 add min/offset and WHERE options to rebuild_zebra This patch reimplement a feature that is on biblibre/master for Koha-community/master It adds 4 parameters: * offset = the ofset of record. Say 1000 to start rebuilding at the 1000th record of your database * min = how many records to export. Say 400 to export only 400 records * WHERE = add a where clause to rebuild only a given itemtype, or anything you want to filter on * l = how many items should be export with biblios. This is a usefull option when you have records with so many items that they can result in a record higher than 99999bytes, that zebra don't like at all Another improvement resulting from ofset & min limit is the rebuild_zebra_sliced.zsh that will be submitted in another patch. _sliced will slice your all database in small chunks, and, if something went wrong for a given slice, will slice the slice, and repeat, until you reach a slice size of 1, showing which record is wrong in your database. Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA CC| |jcamins@cpbibliography.com --- Comment #11 from Jared Camins-Esakov <jcamins@cpbibliography.com> 2012-01-29 14:17:06 UTC --- The script works, but on further reflection it strikes me that the names of the new arguments do not make sense (I corrected "ofset" to "offset"): -offset implies the first record that's being processed, as does -min. Perhaps you could change the arguments to -offset and -length, or something else like that? -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #7379|0 |1 is obsolete| | --- Comment #12 from Paul Poulain <paul.poulain@biblibre.com> 2012-02-01 14:15:36 UTC --- Created attachment 7410 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7410 Bug 7246 add offset/length and where options to rebuild_zebra This patch reimplement a feature that is on biblibre/master for Koha-community/master It adds 4 parameters: * offset = the offset of record. Say 1000 to start rebuilding at the 1000th record of your database * length = how many records to export. Say 400 to export only 400 records * where = add a where clause to rebuild only a given itemtype, or anything you want to filter on * l = how many items should be export with biblios. This is a usefull option when you have records with so many items that they can result in a record higher than 99999bytes, that zebra don't like at all Another improvement resulting from offset & length limit is the rebuild_zebra_sliced.zsh that will be submitted in another patch. rebuild_zebra_sliced will slice your all database in small chunks, and, if something went wrong for a given slice, will slice the slice, and repeat, until you reach a slice size of 1, showing which record is wrong in your database. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|PATCH-Sent (DO NOT USE) |P2 Status|Failed QA |Needs Signoff AssignedTo|dpavlin@rot13.org |paul.poulain@biblibre.com --- Comment #13 from Paul Poulain <paul.poulain@biblibre.com> 2012-02-01 14:17:46 UTC --- (In reply to comment #11)
The script works, but on further reflection it strikes me that the names of the new arguments do not make sense (I corrected "ofset" to "offset"): -offset implies the first record that's being processed, as does -min. Perhaps you could change the arguments to -offset and -length, or something else like that?
You're right. our excuse is that we're french and our english is not always perfect ;-) New patch with offset/length and minor changes in documentation (--param instead of -param in the doc) -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |7286 -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #7410|0 |1 is obsolete| | --- Comment #14 from Jared Camins-Esakov <jcamins@cpbibliography.com> 2012-02-07 13:37:49 UTC --- Created attachment 7470 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7470 Bug 7246 add offset/length and where options to rebuild_zebra This patch reimplement a feature that is on biblibre/master for Koha-community/master It adds 4 parameters: * offset = the offset of record. Say 1000 to start rebuilding at the 1000th record of your database * length = how many records to export. Say 400 to export only 400 records * where = add a where clause to rebuild only a given itemtype, or anything you want to filter on * l = how many items should be export with biblios. This is a usefull option when you have records with so many items that they can result in a record higher than 99999bytes, that zebra don't like at all Another improvement resulting from offset & length limit is the rebuild_zebra_sliced.zsh that will be submitted in another patch. rebuild_zebra_sliced will slice your all database in small chunks, and, if something went wrong for a given slice, will slice the slice, and repeat, until you reach a slice size of 1, showing which record is wrong in your database. Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Ian Walls <koha.sekjal@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #15 from Ian Walls <koha.sekjal@gmail.com> 2012-02-07 15:38:42 UTC --- This patch adds the param for item_limit, as well as the documentation, but no actual support for it. Either the -l param and docs should be removed from this patch, and put in with another that fully implements, or the support for item_limit should be added to this patch. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #7470|0 |1 is obsolete| | --- Comment #16 from Jared Camins-Esakov <jcamins@cpbibliography.com> 2012-02-10 15:15:46 UTC --- Created attachment 7564 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=7564 Bug 7246 add offset/length and where options to rebuild_zebra This patch reimplement a feature that is on biblibre/master for Koha-community/master It adds 4 parameters: * offset = the offset of record. Say 1000 to start rebuilding at the 1000th record of your database * length = how many records to export. Say 400 to export only 400 records * where = add a where clause to rebuild only a given itemtype, or anything you want to filter on Another improvement resulting from offset & length limit is the rebuild_zebra_sliced.zsh that will be submitted in another patch. rebuild_zebra_sliced will slice your all database in small chunks, and, if something went wrong for a given slice, will slice the slice, and repeat, until you reach a slice size of 1, showing which record is wrong in your database. Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> Removed mention of -l option for limiting number of items exported, as requested by QA manager. This can be re-added in a later patch. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Jared Camins-Esakov <jcamins@cpbibliography.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Ian Walls <koha.sekjal@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #17 from Ian Walls <koha.sekjal@gmail.com> 2012-02-16 17:12:03 UTC --- Thank you, Jared, for removing the unused param. We can add it back in at a later time. Marking Passed QA. -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7246 Paul Poulain <paul.poulain@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master Version|master |rel_3_8 --- Comment #18 from Paul Poulain <paul.poulain@biblibre.com> 2012-02-17 10:01:37 UTC --- Patch pushed. Chris_n, this is an ENH that can safely be added to 3.6 as it add a new feature and don't change anything to what is existing. However, unless it's properly documented, it may be missed by most ppl. So i've set it for 3.8, but if you want to commit it to 3.6, please add a comment here -- Configure bugmail: http://bugs.koha-community.org/bugzilla3/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org