[Bug 43142] New: Elasticsearch number_of_replicas should be driven by a system preference
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 Bug ID: 43142 Summary: Elasticsearch number_of_replicas should be driven by a system preference Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: tomascohen@gmail.com QA Contact: testopia@bugs.koha-community.org Target Milestone: --- Koha's index_config.yaml hardcodes index.number_of_replicas: 1 and index.number_of_shards: 5. On single-node clusters (the vast majority of Koha deployments), this causes a permanently yellow cluster health status because replica shards can never be assigned. The number_of_replicas setting is applied at index creation time during rebuild_elasticsearch.pl. Since this is entirely within Koha's control, it should be configurable via a system preference. Proposed solution: - Add a new system preference ElasticsearchReplicas (default: 0) - rebuild_elasticsearch.pl and any index creation code should read this preference and override the value from index_config.yaml - Default of 0 is sensible because: - Most deployments are single-node - Replicas are only useful with multiple ES nodes - Organizations that run multi-node clusters can set it to 1 or higher - The setting takes effect on rebuild, so changing it is straightforward This also eliminates the confusing yellow cluster health on fresh installs and KTD development environments. Note: The index_config.yaml default could also be changed to 0, with the syspref allowing organizations to increase it. Either approach works, but the syspref is the important part for operational control. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |tomascohen@gmail.com |ity.org | CC| |nick@bywatersolutions.com, | |tomascohen@gmail.com Status|NEW |ASSIGNED -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Trivial patch Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 --- Comment #1 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 202242 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202242&action=edit Bug 43142: Make Elasticsearch number_of_replicas driven by a syspref Adds ElasticsearchReplicas system preference (default: 0) which overrides the index.number_of_replicas value from index_config.yaml when creating indexes. The previous hardcoded default of 1 replica caused permanently yellow cluster health on single-node deployments (the majority of Koha installations) because replica shards cannot be assigned without additional nodes. With the default of 0, single-node clusters get green health status immediately after reindexing. Multi-node deployments can set the preference to 1 or higher as needed. Takes effect on next reindex (rebuild_elasticsearch.pl). Test plan: 1. Apply patch 2. Run: $ ktd --shell k$ cd /kohadevbox/koha && perl installer/data/mysql/updatedatabase.pl => SUCCESS: Added new system preference 'ElasticsearchReplicas' 3. Verify default value: k$ koha-mysql kohadev -e "SELECT * FROM systempreferences WHERE variable='ElasticsearchReplicas';" => SUCCESS: value = 0 4. Rebuild indexes: k$ perl misc/search_tools/rebuild_elasticsearch.pl --delete 5. Check cluster health: k$ curl -s http://es:9200/_cluster/health | python3 -m json.tool => SUCCESS: status is green, unassigned_shards is 0 6. Sign off :-D Assisted-by: Sonnet 4 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=43137 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202242|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 --- Comment #2 from David Nind <david@davidnind.com> --- Created attachment 202279 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202279&action=edit Bug 43142: Make Elasticsearch number_of_replicas driven by a syspref Adds ElasticsearchReplicas system preference (default: 0) which overrides the index.number_of_replicas value from index_config.yaml when creating indexes. The previous hardcoded default of 1 replica caused permanently yellow cluster health on single-node deployments (the majority of Koha installations) because replica shards cannot be assigned without additional nodes. With the default of 0, single-node clusters get green health status immediately after reindexing. Multi-node deployments can set the preference to 1 or higher as needed. Takes effect on next reindex (rebuild_elasticsearch.pl). Test plan: 1. Apply patch 2. Run: $ ktd --shell k$ cd /kohadevbox/koha && perl installer/data/mysql/updatedatabase.pl => SUCCESS: Added new system preference 'ElasticsearchReplicas' 3. Verify default value: k$ koha-mysql kohadev -e "SELECT * FROM systempreferences WHERE variable='ElasticsearchReplicas';" => SUCCESS: value = 0 4. Rebuild indexes: k$ perl misc/search_tools/rebuild_elasticsearch.pl --delete 5. Check cluster health: k$ curl -s http://es:9200/_cluster/health | python3 -m json.tool => SUCCESS: status is green, unassigned_shards is 0 6. Sign off :-D Assisted-by: Sonnet 4 (Anthropic) Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com Text to go in the| |This enhancement adds a new release notes| |local use system | |preference, | |ElasticsearchReplicas, with | |the default value set to 0. | | | |Technical details: | | | |This overrides the | |index.number_of_replicas | |value from | |index_config.yaml | |when creating indexes. | | | |The previous hardcoded | |default of 1 replica caused | |permanently yellow | |cluster health on | |single-node deployments | |(the majority of Koha | |installations) because | |replica shards cannot be | |assigned without | |additional nodes. | | | |With the default of 0, | |single-node clusters get | |green health status | |immediately after | |reindexing. Multi-node | |deployments can set the | |preference to 1 or higher | |as needed. | | | |Changes take effect on the | |next reindex. | | | |The index_config.yaml | |default could also be | |changed to 0, with the | |system preference allowing | |organizations to increase | |it. Either approach works, | |but the system preference | |is the important part for | |operational control. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 --- Comment #3 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- I forgot to commit the `searchging.pref` changes. Booo -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 --- Comment #4 from David Nind <david@davidnind.com> --- (In reply to Tomás Cohen Arazi (tcohen) from comment #3)
I forgot to commit the `searchging.pref` changes. Booo
I wondered! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202279|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 --- Comment #5 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 202395 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202395&action=edit Bug 43142: Make Elasticsearch number_of_replicas driven by a syspref Adds ElasticsearchReplicas system preference (default: 0) which overrides the index.number_of_replicas value from index_config.yaml when creating indexes. The previous hardcoded default of 1 replica caused permanently yellow cluster health on single-node deployments (the majority of Koha installations) because replica shards cannot be assigned without additional nodes. With the default of 0, single-node clusters get green health status immediately after reindexing. Multi-node deployments can set the preference to 1 or higher as needed. Takes effect on next reindex (rebuild_elasticsearch.pl). Test plan: 1. Apply patch 2. Run: $ ktd --shell k$ cd /kohadevbox/koha && perl installer/data/mysql/updatedatabase.pl => SUCCESS: Added new system preference 'ElasticsearchReplicas' 3. Verify default value: k$ koha-mysql kohadev -e "SELECT * FROM systempreferences WHERE variable='ElasticsearchReplicas';" => SUCCESS: value = 0 4. Rebuild indexes: k$ perl misc/search_tools/rebuild_elasticsearch.pl --delete 5. Check cluster health: k$ curl -s http://es:9200/_cluster/health | python3 -m json.tool => SUCCESS: status is green, unassigned_shards is 0 6. Sign off :-D Assisted-by: Sonnet 4 (Anthropic) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202395|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 --- Comment #6 from David Nind <david@davidnind.com> --- Created attachment 202441 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202441&action=edit Bug 43142: Make Elasticsearch number_of_replicas driven by a syspref Adds ElasticsearchReplicas system preference (default: 0) which overrides the index.number_of_replicas value from index_config.yaml when creating indexes. The previous hardcoded default of 1 replica caused permanently yellow cluster health on single-node deployments (the majority of Koha installations) because replica shards cannot be assigned without additional nodes. With the default of 0, single-node clusters get green health status immediately after reindexing. Multi-node deployments can set the preference to 1 or higher as needed. Takes effect on next reindex (rebuild_elasticsearch.pl). Test plan: 1. Apply patch 2. Run: $ ktd --shell k$ cd /kohadevbox/koha && perl installer/data/mysql/updatedatabase.pl => SUCCESS: Added new system preference 'ElasticsearchReplicas' 3. Verify default value: k$ koha-mysql kohadev -e "SELECT * FROM systempreferences WHERE variable='ElasticsearchReplicas';" => SUCCESS: value = 0 4. Rebuild indexes: k$ perl misc/search_tools/rebuild_elasticsearch.pl --delete 5. Check cluster health: k$ curl -s http://es:9200/_cluster/health | python3 -m json.tool => SUCCESS: status is green, unassigned_shards is 0 6. Sign off :-D Assisted-by: Sonnet 4 (Anthropic) Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43142 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the|This enhancement adds a new |This enhancement adds a new release notes|local use system |Elasticsearch related |preference, |system preference, |ElasticsearchReplicas, with |ElasticsearchReplicas, with |the default value set to 0. |the default value set to 0. | | |Technical details: |For Koha instances with | |single-node deployments of |This overrides the |Elasticsearch (the majority |index.number_of_replicas |of Koha installations), no |value from |changes should be required. |index_config.yaml | |when creating indexes. |This setting should be | |reviewed where a more |The previous hardcoded |complex Elasticsearch |default of 1 replica caused |configuration is in place. |permanently yellow | |cluster health on |Technical details: |single-node deployments | |(the majority of Koha |This overrides the |installations) because |index.number_of_replicas |replica shards cannot be |value from |assigned without |index_config.yaml when |additional nodes. |creating indexes. | | |With the default of 0, |The previous hardcoded |single-node clusters get |default of 1 replica caused |green health status |permanently yellow cluster |immediately after |health on single-node |reindexing. Multi-node |deployments (the majority |deployments can set the |of Koha installations) |preference to 1 or higher |because replica shards |as needed. |cannot be assigned without | |additional nodes. |Changes take effect on the | |next reindex. |With the default of 0, | |single-node clusters get |The index_config.yaml |green health status |default could also be |immediately after |changed to 0, with the |reindexing. Multi-node |system preference allowing |deployments can set the |organizations to increase |preference to 1 or higher |it. Either approach works, |as needed. |but the system preference | |is the important part for |Changes take effect on the |operational control. |next reindex. | | | |The index_config.yaml | |default could also be | |changed to 0, with the | |system preference allowing | |organizations to increase | |it. Either approach works, | |but the system preference | |is the important part for | |operational control. -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org