[Bug 28268] New: Improve memory usage when indexing authorities in Elasticsearch
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Bug ID: 28268 Summary: Improve memory usage when indexing authorities in Elasticsearch Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: ere.maijala@helsinki.fi Currently indexing authorities in Elasticsearch creates a record set of all reocrds that includes the MARCXML records. This can use a huge amount of memory and prevent the indexing altogether. We should do the same as we do with biblios in that the full record set only contains the authority id's. Further on, it would make sense to process the record in similar way to bug 27584, but that's another issue. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |ere.maijala@helsinki.fi |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart+koha@gmail. | |com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 --- Comment #1 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- I am not sure I understand. It's an iterator, we are not going to load all the records in memory. The max we expect is the --commit param passed to the script (default 5000). Waiting for your patch :) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 --- Comment #2 from Ere Maijala <ere.maijala@helsinki.fi> --- Created attachment 120329 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=120329&action=edit Bug 28268: Improve memory usage when indexing authorities in Elasticsearch Retrieves the complete records one by one to avoid huge memory usage. Note that this removes the call to GuessAuthTypeCode, but it is done later in Koha::SearchEngine::Elasticsearch::marc_records_to_documents (and was never done if you asked to index a single record with --authid parameter). Test plan: 1. Apply patch 2. Reindex authorities: perl misc/search_tools/rebuild_elasticsearch.pl -a -d -v 3. Check that indexing completed successfully and results are correct. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #120329|0 |1 is obsolete| | --- Comment #3 from Ere Maijala <ere.maijala@helsinki.fi> --- Created attachment 120330 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=120330&action=edit Bug 28268: Improve memory usage when indexing authorities in Elasticsearch Retrieves the complete records one by one to avoid huge memory usage. Note that this removes the call to GuessAuthTypeCode, but it is done later in Koha::SearchEngine::Elasticsearch::marc_records_to_documents (and was never done if you asked to index a single record with --authid parameter). Test plan: 1. Apply patch 2. Reindex authorities: perl misc/search_tools/rebuild_elasticsearch.pl -a -d -v 3. Check that indexing completed successfully and results are correct. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 --- Comment #4 from Ere Maijala <ere.maijala@helsinki.fi> --- This patch changes the mechanism so that the callback used to retrieve a single record retrieves authtypecode and marcxml fields. Well, actually it delegates the work to get_from_authid. As a result the record set retrieved from database only contains authid, which consumes quite a bit less memory than all the fields together. This also mirrors the method used in get_all_biblios_iterator. Surprisingly this seems to be also faster than the old mechanism. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- URL| |https://lists.katipo.co.nz/ | |pipermail/koha/2021-April/0 | |56140.html -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 jmbroust <jean-manuel.broust@univ-lyon2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jean-manuel.broust@univ-lyo | |n2.fr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #120330|0 |1 is obsolete| | --- Comment #5 from Aleisha Amohia <aleisha@catalyst.net.nz> --- Created attachment 120424 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=120424&action=edit Bug 28268: Improve memory usage when indexing authorities in Elasticsearch Retrieves the complete records one by one to avoid huge memory usage. Note that this removes the call to GuessAuthTypeCode, but it is done later in Koha::SearchEngine::Elasticsearch::marc_records_to_documents (and was never done if you asked to index a single record with --authid parameter). Test plan: 1. Apply patch 2. Reindex authorities: perl misc/search_tools/rebuild_elasticsearch.pl -a -d -v 3. Check that indexing completed successfully and results are correct. Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com --- Comment #6 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 120865 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=120865&action=edit performance comparison Three runs before and after the patch, 53927 auth records - machine generated from about ~30k bibs Command I ran: /usr/bin/time -v perl misc/search_tools/rebuild_elasticsearch.pl -v -d -a -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #7 from Nick Clemens <nick@bywatersolutions.com> --- This does seem to decrease memory, but I found it tended to run a bit longer time wise The code is simple, I don't like the 'while(1)' Could the fetch of the next row be moved into the while? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 --- Comment #8 from Ere Maijala <ere.maijala@helsinki.fi> --- Nick, it's the same setup as in Biblio's iterator. If that's to be changed, I'd rather do it separately. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #120424|0 |1 is obsolete| | --- Comment #9 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 120866 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=120866&action=edit Bug 28268: Improve memory usage when indexing authorities in Elasticsearch Retrieves the complete records one by one to avoid huge memory usage. Note that this removes the call to GuessAuthTypeCode, but it is done later in Koha::SearchEngine::Elasticsearch::marc_records_to_documents (and was never done if you asked to index a single record with --authid parameter). Test plan: 1. Apply patch 2. Reindex authorities: perl misc/search_tools/rebuild_elasticsearch.pl -a -d -v 3. Check that indexing completed successfully and results are correct. Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |21.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 --- Comment #10 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Pushed to master for 21.05, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Status|Pushed to master |Pushed to stable Version(s)|21.05.00 |21.05.00,20.11.06 released in| | --- Comment #11 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 20.11.x for 20.11.06 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew@bywatersolutions.com --- Comment #12 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Doesn't apply cleanly to 20.05, please rebase if needed -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28268 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |33205 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33205 [Bug 33205] (Bug 28268 follow-up) Method call $row->authid inside quotes - produces meaningless warning -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org