[Bug 26996] New: Elasticsearch: Multiprocess reindexing sometimes doesn't reindex all records
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Bug ID: 26996 Summary: Elasticsearch: Multiprocess reindexing sometimes doesn't reindex all records Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: minor Priority: P5 - low Component: Searching - Zebra Assignee: koha-bugs@lists.koha-community.org Reporter: bjorn.nylen@ub.lu.se We've discovered that rebuild_elasticsearch.pl will not index all records in some cases using multiple cpu's. Sometimes one (or more) child proc. will loop through their records without ever committing them to ES. Symptoms are that the process will work faster than others, not log "Committing xxx records" and consume ever growing amount of memory. The problem appears to be in the fudging of commmit sizes for the childs. It generates a float which will in some cases never be == 0 as the buffer counter is decreased. This set of param will fail for me everytime ./rebuild_elasticsearch -v -b -p 2 -c 400 Will uplaod a patch shortly. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Björn Nylén <bjorn.nylen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |bjorn.nylen@ub.lu.se |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Björn Nylén <bjorn.nylen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|Searching - Zebra |Searching - Elasticsearch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Björn Nylén <bjorn.nylen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Change sponsored?|--- |Sponsored -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Björn Nylén <bjorn.nylen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Change sponsored?|Sponsored |--- -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 --- Comment #1 from Björn Nylén <bjorn.nylen@ub.lu.se> --- Created attachment 113482 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=113482&action=edit Truncate commit size to make sure it's an integer Patch to truncate commit size to be sure it's an integer To test you must find a set of circumstances that causes the issue. For me: 1. Run: ./rebuild_elasticsearch -v -b -p 2 -c 400 2. Note that only one process is logging "Committing xxx records..." 3. Kill processes. 4. Apply patch. 5. Repeat 1 6. Note that only both processes are logging "Committing xxx records..." -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Björn Nylén <bjorn.nylen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Small patch Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Joonas Kylmälä <joonas.kylmala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joonas.kylmala@helsinki.fi Status|Needs Signoff |Failed QA --- Comment #2 from Joonas Kylmälä <joonas.kylmala@helsinki.fi> --- Thanks, this fixes the problem indeed. Few things to improve on the patch: - In the commit title describe what the fix does, not what the problem is. E.g. "Convert Elasticsearch indexer commit buffer size to int" and in the body describe that it fixes the buffer counter not resetting and the reason why it fixes it. - In the commit title use "Bug XXXX: YYY" format which is used by every other commit we have (now there is the "-" character instead). - Add whitespace between the multiplication and parentheses. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Björn Nylén <bjorn.nylen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #113482|0 |1 is obsolete| | --- Comment #3 from Björn Nylén <bjorn.nylen@ub.lu.se> --- Created attachment 114227 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=114227&action=edit Truncate commit size to make sure it's an integer - QA update New patch with suggested QA changes. Moved the truncation to a separeate line to make it more obvious. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Björn Nylén <bjorn.nylen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Joonas Kylmälä <joonas.kylmala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Joonas Kylmälä <joonas.kylmala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #114227|0 |1 is obsolete| | --- Comment #4 from Joonas Kylmälä <joonas.kylmala@helsinki.fi> --- Created attachment 114232 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=114232&action=edit Bug 26996: Convert Elasticsearch indexer commit buffer size to integer When multithreaded indexing is used, the commit size for children are spread out resulting in them being of type float. When records are processed and the commit counter decreased it may then never reach *exactly* 0. This means records are never commited. This patch makes sure the counter is an integer to avoid the problem. To test you must find a set of circumstances that causes the issue. For me: 1. Run: ./rebuild_elasticsearch -v -b -p 2 -c 400 2. Note that only one process is logging "Committing xxx records..." 3. Kill processes. 4. Apply patch. 5. Repeat 1 6. Note that both processes are logging "Committing xxx records..." Sponsored-by: Lund University Library Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #114232|0 |1 is obsolete| | --- Comment #5 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 114679 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=114679&action=edit Bug 26996: Convert Elasticsearch indexer commit buffer size to integer When multithreaded indexing is used, the commit size for children are spread out resulting in them being of type float. When records are processed and the commit counter decreased it may then never reach *exactly* 0. This means records are never commited. This patch makes sure the counter is an integer to avoid the problem. To test you must find a set of circumstances that causes the issue. For me: 1. Run: ./rebuild_elasticsearch -v -b -p 2 -c 400 2. Note that only one process is logging "Committing xxx records..." 3. Kill processes. 4. Apply patch. 5. Repeat 1 6. Note that both processes are logging "Committing xxx records..." Sponsored-by: Lund University Library Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com --- Comment #6 from Nick Clemens <nick@bywatersolutions.com> --- In testing it did seem as if the records were committed in the end, caught by the final index of remaining records, but very not ideal, especially in a large system -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |21.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 --- Comment #7 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 21.05, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|21.05.00 |21.05.00,20.11.02 released in| | CC| |fridolin.somers@biblibre.co | |m Status|Pushed to master |Pushed to stable --- Comment #8 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 20.11.x for 20.11.02 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew@bywatersolutions.com Status|Pushed to stable |Pushed to oldstable Version(s)|21.05.00,20.11.02 |21.05.00,20.11.02,20.05.08 released in| | --- Comment #9 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Pushed to 20.05.x for 20.05.08 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |victor@tuxayo.net Status|Pushed to oldstable |RESOLVED --- Comment #10 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Not backported to oldoldstable (19.11.x). Feel free to ask if it's needed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26996 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |caroline.cyr-la-rose@inlibr | |o.com -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org