[Bug 40890] New: Make batch_size configurable for koha-es-indexer
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Bug ID: 40890 Summary: Make batch_size configurable for koha-es-indexer Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: tomascohen@gmail.com QA Contact: testopia@bugs.koha-community.org The script allows setting a custom batch_size e.g. `koha-es-indexer --batch_size 100 --restart library` but it cannot be set permanently. So a reboot or a naive restart will set it back to 10. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |tomascohen@gmail.com |ity.org | CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle@bywatersolutions.com, | |mtj@kohaaloha.com, | |tristin.stagg@bywatersoluti | |ons.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Patch complexity|--- |String patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 --- Comment #1 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 187014 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=187014&action=edit Bug 40890: Make batch_size configurable for koha-es-indexer This patch adds a new configuration entry in `koha-conf.xml` for making it possible to set a different `batch_size` for `koha-es-indexer`. The current implementation has a default of *10* that can be overridden using the CLI parameter `--batch_size N`. Unfortunately, if you restart the server or the service without passing the parameter, it will be set back to 10. This patch solves that by adding a new config entry and making the script use it. The passed CLI parameter will keep overridding the configured value. To test: 1. Run: $ ktd --shell k$ koha-es-indexer --restart --batch_size 100 kohadev k$ ps -ef | grep elastic => SUCCESS: No errors. Notice '100' is passed to the called perl script 2. Run: k$ koha-es-indexer --restart kohadev k$ ps -ef | grep elastic => FAIL: Notice '10' is the new value 3. Apply this patch 4. Run: k$ reset_all 5. Repeat 1 and 2 => SUCCESS: '100' is kept. 6. Try standalone `--start` too 7. Sign off :-D -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|String patch |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |A new configuration entry release notes| |has been introduced in | |`koha-conf.xml` to control | |how many Elasticsearch | |update tasks the indexer | |processes per iteration. | | | |Previously, the indexer | |defaulted to a batch size | |of 10. While the | |`koha-es-indexer` command | |supported overriding this | |with the `--batch_size` | |parameter, there was no way | |to set a different default | |value. | | | |This change is particularly | |beneficial for large sites, | |where the indexer may | |otherwise struggle to keep | |pace with the update queue | |during peak activity. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #187014|0 |1 is obsolete| | --- Comment #2 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 188996 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=188996&action=edit Bug 40890: Make batch_size configurable for koha-es-indexer This patch adds a new configuration entry in `koha-conf.xml` for making it possible to set a different `batch_size` for `koha-es-indexer`. The current implementation has a default of *10* that can be overridden using the CLI parameter `--batch_size N`. Unfortunately, if you restart the server or the service without passing the parameter, it will be set back to 10. This patch solves that by adding a new config entry and making the script use it. The passed CLI parameter will keep overridding the configured value. To test: 1. Run: $ ktd --shell k$ koha-es-indexer --restart --batch_size 100 kohadev k$ ps -ef | grep elastic => SUCCESS: No errors. Notice '100' is passed to the called perl script 2. Run: k$ koha-es-indexer --restart kohadev k$ ps -ef | grep elastic => FAIL: Notice '10' is the new value 3. Apply this patch 4. Run: k$ reset_all 5. Repeat 1 and 2 => SUCCESS: '100' is kept. 6. Try standalone `--start` too 7. Sign off :-D Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 --- Comment #3 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 188997 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=188997&action=edit Bug 40890: Move fetching of batch size to within the instance loop Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@openfifth.c | |o.uk QA Contact|testopia@bugs.koha-communit |martin.renvoize@openfifth.c |y.org |o.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #188996|0 |1 is obsolete| | --- Comment #4 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 189143 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189143&action=edit Bug 40890: Make batch_size configurable for koha-es-indexer This patch adds a new configuration entry in `koha-conf.xml` for making it possible to set a different `batch_size` for `koha-es-indexer`. The current implementation has a default of *10* that can be overridden using the CLI parameter `--batch_size N`. Unfortunately, if you restart the server or the service without passing the parameter, it will be set back to 10. This patch solves that by adding a new config entry and making the script use it. The passed CLI parameter will keep overridding the configured value. To test: 1. Run: $ ktd --shell k$ koha-es-indexer --restart --batch_size 100 kohadev k$ ps -ef | grep elastic => SUCCESS: No errors. Notice '100' is passed to the called perl script 2. Run: k$ koha-es-indexer --restart kohadev k$ ps -ef | grep elastic => FAIL: Notice '10' is the new value 3. Apply this patch 4. Run: k$ reset_all 5. Repeat 1 and 2 => SUCCESS: '100' is kept. 6. Try standalone `--start` too 7. Sign off :-D Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #188997|0 |1 is obsolete| | --- Comment #5 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 189144 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=189144&action=edit Bug 40890: Move fetching of batch size to within the instance loop Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |25.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 --- Comment #6 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Nice work everyone! Pushed to main for 25.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Evelyn Hartline <evelyn@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |evelyn@bywatersolutions.com --- Comment #7 from Evelyn Hartline <evelyn@bywatersolutions.com> --- Could this be backported to 25.05? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 --- Comment #8 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- I'd hold off on that.. right now this has taken down some servers where we backported it and I'm investigating -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 --- Comment #9 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- OK.. it was a bad backport to 24.11.x for me.. feel free to go ahead -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42484 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42484 [Bug 42484] dead and misleading code remained in koha-es-indexer after 40890 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40890 Laura Escamilla <Laura.escamilla@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |Laura.escamilla@bywatersolu | |tions.com Status|Pushed to main |Needs documenting --- Comment #10 from Laura Escamilla <Laura.escamilla@bywatersolutions.com> --- Not backported to 25.05.x as this is an enhancement but I’m happy to revisit if needed. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org