[Bug 18948] New: Elasticsearch - Reindexes should use aliases to avoid down time while reindexing
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Bug ID: 18948 Summary: Elasticsearch - Reindexes should use aliases to avoid down time while reindexing Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: nick@bywatersolutions.com Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org This patchset aims to add the ability to leverage the ES alias feature during a reindex - when doing a full reindex or a deletion the indexer will look for an existing index under the alias - if not found it will create a new one, if found it will index into a new index and point the alias there when done and then delete the old index. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 65086 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65086&action=edit Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing To test: 1 - Delete your current indices for ES (if any) 2 - Reindex your records 3 - curl -XGET 'localhost:9200/koha_kohadev?pretty' 4 - Note indices are now of form koha_kohadev_########## 5 - The numbers there are a result of appending 'time' command to name 6 - Ensure searching works as before the patches 7 - Reindex your records with a commit setting of 1 (to slow things down) you may need a large numebr fo records 8 - Ensure searching works during reindex 9 - Ensure reindexing completes successfully 10 - curl -XGET 'localhost:9200/koha_kohadev?pretty' 11 - There should only be one index each for biblios and authorities ** - Note number in above 12 - Perform a partial reindex by passing a biblionumber to the command 13 - Curl the indices again and ensure number/name has not changed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org, | |kyle@bywatersolutions.com, | |ldjamison@marywood.edu, | |tomascohen@gmail.com --- Comment #2 from Nick Clemens <nick@bywatersolutions.com> --- TODO: 1 - Ensure if records are indexed incrementally before first rebuild that an alias is used 2 - Unit tests Other comments welcome, this was first round proof of concept -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 --- Comment #3 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 65147 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65147&action=edit Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing Refactoring of rebuild_elastic_search.pl to move code into Koha::SearchEngine::Elasticsearch::Indexer -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #65086|0 |1 is obsolete| | --- Comment #4 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 65159 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=65159&action=edit Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing To test: 1 - Delete your current indices for ES (if any) 2 - Reindex your records 3 - curl 'localhost:9200/_cat/indices?v' 4 - Note indices are now of form koha_kohadev_########## 5 - The numbers there are a result of appending 'time' command to name 6 - Ensure searching works as before the patches 7 - Reindex your records with a commit setting of 1 (to slow things down) you may need a large numebr fo records 8 - Ensure searching works during reindex 9 - Ensure reindexing completes successfully 10 - curl 'localhost:9200/_cat/indices?v' 11 - There should only be one index each for biblios and authorities ** - Note number in above 12 - Perform a partial reindex by passing a biblionumber to the command 13 - Curl the indices again and ensure number/name has not changed 14 - Delete all indexes curl -XDELETE 'localhost:9200/koha_kohadev_*?pretty' 15 - Edit a known biblio: http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=1 16 - Add a word to the title "potato" 17 - Save the record 18 - Search for potato 19 - You should arrive at the detail page for that record 20 - Confirm your index is of form koha_kohadev_########## -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 --- Comment #5 from Nick Clemens <nick@bywatersolutions.com> --- Needs unit tests -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |18950 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18950 [Bug 18950] Elasticsearch - Add password access for admin functions -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|Architecture, internals, |Searching - Elasticsearch |and plumbing | Summary|Elasticsearch - Reindexes |Reindexing should use |should use aliases to avoid |aliases to avoid down time |down time while reindexing | QA Contact|testopia@bugs.koha-communit | |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 David Bourgault <david.bourgault@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david.bourgault@inlibro.com Status|Needs Signoff |Patch doesn't apply --- Comment #6 from David Bourgault <david.bourgault@inlibro.com> --- Patch does not apply, SHA1 error for Indexer.pm -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #65147|0 |1 is obsolete| | Attachment #65159|0 |1 is obsolete| | --- Comment #7 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 70815 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=70815&action=edit Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing Refactoring of rebuild_elastic_search.pl to move code into Koha::SearchEngine::Elasticsearch::Indexer Current status: Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 --- Comment #8 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 70816 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=70816&action=edit Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing To test: 1 - Delete your current indices for ES (if any) 2 - Reindex your records 3 - curl 'localhost:9200/_cat/indices?v' 4 - Note indices are now of form koha_kohadev_########## 5 - The numbers there are a result of appending 'time' command to name 6 - Ensure searching works as before the patches 7 - Reindex your records with a commit setting of 1 (to slow things down) you may need a large numebr fo records 8 - Ensure searching works during reindex 9 - Ensure reindexing completes successfully 10 - curl 'localhost:9200/_cat/indices?v' 11 - There should only be one index each for biblios and authorities ** - Note number in above 12 - Perform a partial reindex by passing a biblionumber to the command 13 - Curl the indices again and ensure number/name has not changed 14 - Delete all indexes curl -XDELETE 'localhost:9200/koha_kohadev_*?pretty' 15 - Edit a known biblio: http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=1 16 - Add a word to the title "potato" 17 - Save the record 18 - Search for potato 19 - You should arrive at the detail page for that record 20 - Confirm your index is of form koha_kohadev_########## -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |ASSIGNED --- Comment #9 from Nick Clemens <nick@bywatersolutions.com> --- Ugly rebase, still needs unit tests and testing to confirm it works -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ere.maijala@helsinki.fi --- Comment #10 from Ere Maijala <ere.maijala@helsinki.fi> --- I'd like to see an option that uses the existing index also when reindexing all records. In my experience there are cases where you want to see the changes as soon as possible and not wait until the whole indexing process is complete. Testing indexing is just one of them but pretty important during development. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtj@kohaaloha.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |severine.queune@bulac.fr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 --- Comment #11 from Ere Maijala <ere.maijala@helsinki.fi> --- Additionally, if aliases are used, the there should be a mechanism to take care of indexing all the changes that were made in staff UI or such during the reindexing process. Otherwise they'd end up existing only in the old index that gets deleted, right? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEW Assignee|nick@bywatersolutions.com |koha-bugs@lists.koha-commun | |ity.org --- Comment #12 from Nick Clemens <nick@bywatersolutions.com> --- Not working on this currently, leaving it for anyone who wants to look -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 claire.hernandez@biblibre.com <claire.hernandez@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |claire.hernandez@biblibre.c | |om -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 --- Comment #13 from claire.hernandez@biblibre.com <claire.hernandez@biblibre.com> --- +1 it would be great to use alias for not loosing search results during full reindexing \o/ -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |domm@plix.at --- Comment #14 from Thomas Klausner <domm@plix.at> --- good idea, but I think it makes no sense to fix the old patches from 2018. I also wonder if this isn't something that can be handled directly in ES (via some API calls to set/link the alias), and maybe a "simple" param to rebuild_elasticsearch.pl to override the ES index name on the commandline (so when indexing we can use `--index-into new-index` and when it's done do a ES API call to link `index` from `old-index` to `new-index` -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 HKS3 Tadeusz Sośnierz <tadeusz@sosnierz.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tadeusz@sosnierz.com Assignee|koha-bugs@lists.koha-commun |tadeusz@sosnierz.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 --- Comment #15 from HKS3 Tadeusz Sośnierz <tadeusz@sosnierz.com> --- Created attachment 196156 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=196156&action=edit Bug 18948: (WIP) Use ElasticSearch aliases for zero-downtime reindexing This switches SearchEngine::ElasticSearch to operate on aliases rather than the indexes directly most of the time. We only touch indexes themselves when (re)creating them. When rebuilding an index, a new one is created (with a semi-unique name) written to from that point on (by regular Koha operations), and swapped with the old one once the rebuild is complete, giving us zero-downtime rebuilds. The option to delete the index is now removed, as I can think of no reason to use it: we can have multiple indexes live at the same time and all of them get searched. Known issues: - `index_name` is still used as a name in the code, to now mean "alias name" most of the time. This should change before this ships. - The real index names are the alias name trailed by _<timestamp>. In code this is called "colour", since I initially planned to call them "red" and "blue" or something similar. The timestamp has an added bonus of telling us when each index was created, which may be useful when investigating issues, but we may want to go for something else (and call it differently). This is a work in progress, but complete enough to be discussable. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18948 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=43584 CC| |martin.renvoize@openfifth.c | |o.uk -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org