[Bug 19502] New: Elasticsearch - result sets limited to 10000
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Bug ID: 19502 Summary: Elasticsearch - result sets limited to 10000 Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com With the new feature to jump to a last page of search results it is apparent that we can't do this with our current use of ES. The standard limit is 10000 results, it can be increased at the cost of memory - need to look into 'Scroll' or 'Search after' options https://www.elastic.co/guide/en/elasticsearch/reference/5.5/index-modules.ht... index.max_result_window The maximum value of from + size for searches to this index. Defaults to 10000. Search requests take heap memory and time proportional to from + size and this limits that memory. See Scroll or Search After for a more efficient alternative to raising this. https://www.elastic.co/guide/en/elasticsearch/reference/5.6/search-request-s... https://www.elastic.co/guide/en/elasticsearch/reference/5.6/search-request-s... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |severine.queune@bulac.fr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 73020 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=73020&action=edit Bug 19502: Limit pagination to first 10000 results when using ES This patch is to avoid hitting an error page. We should eventually make the max number returned configurable for ES. To test: 1 - Have Koha running ES with 10,000+ records 2 - Search for '*' 3 - Click 'Last' to view last page of results 4 - 'Cannot perform search' error 5 - Apply patch 6 - Search again 7 - View 'Last' page 8 - No error, you go to the last of 10000 9 - Note the warning above the pagination buttons -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | CC| |alex.arnaud@biblibre.com, | |claire.hernandez@biblibre.c | |om, | |gaetan.boisson@biblibre.com | |, nicolas.legrand@bulac.fr -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #73020|0 |1 is obsolete| | --- Comment #2 from Séverine Queune <severine.queune@bulac.fr> --- Created attachment 73024 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=73024&action=edit Bug 19502: Limit pagination to first 10000 results when using ES This patch is to avoid hitting an error page. We should eventually make the max number returned configurable for ES. To test: 1 - Have Koha running ES with 10,000+ records 2 - Search for '*' 3 - Click 'Last' to view last page of results 4 - 'Cannot perform search' error 5 - Apply patch 6 - Search again 7 - View 'Last' page 8 - No error, you go to the last of 10000 9 - Note the warning above the pagination buttons Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #3 from Séverine Queune <severine.queune@bulac.fr> --- Works as described for both staff client and opac. Thanks Nick ! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #4 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 74587 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=74587&action=edit Bug 19520: (follow-up) Pass parameters to avoid making templates depend on search engine -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Elasticsearch - result sets |Result sets limited to |limited to 10000 |10000 CC| |jonathan.druart@bugs.koha-c | |ommunity.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact| |julian.maurice@biblibre.com CC| |julian.maurice@biblibre.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #5 from Julian Maurice <julian.maurice@biblibre.com> --- I think we can do better than hardcoding the ES default value for max_result_window. We can retrieve the actual setting from ES, so that anyone can adjust this setting depending on their needs. I'm currently preparing a patch that will do that. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #6 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 75816 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=75816&action=edit Bug 19502: Retrieve index.max_result_window from ES This avoid hardcoding '10000' in two different places and allow users to adjust this setting. Also, this patch fixes a bug when the search return less than 10000 results Test plan: 1. Do a search that returns 10000+ records. 2. Note the warning above the pagination buttons 3. Go to the last page, no error 4. Change the ES setting: curl -XPUT http://elasticsearch/koha_master_biblios/_settings -d \ '{"index": {"max_result_window": 20000}}' 5. Do another search that returns more than 10000 but less than 20000 6. Note that the warning does not show up 7. Go to the last page, still no error -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|julian.maurice@biblibre.com | --- Comment #7 from Julian Maurice <julian.maurice@biblibre.com> --- Nick, can you take a look at the last patch please ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #8 from Alex Arnaud <alex.arnaud@biblibre.com> --- Comment on attachment 74587 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=74587 Bug 19520: (follow-up) Pass parameters to avoid making templates depend on search engine Review of attachment 74587: --> (https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=19502&attachment=74587) ----------------------------------------------------------------- ::: catalogue/search.pl @@ +677,4 @@
last_page_offset => $last_page_offset, previous_page_offset => $previous_page_offset) unless $pages < 2; $template->param( next_page_offset => $next_page_offset) unless $pages eq $current_page_number; + warn "topage $hits_to_paginate";
Debug thing ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #9 from Julian Maurice <julian.maurice@biblibre.com> --- (In reply to Alex Arnaud from comment #8)
Debug thing ?
Yes. It can be removed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #10 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 76371 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76371&action=edit Bug 19502: Remove debug warn -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #11 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 76372 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76372&action=edit Bug 19502: Add POD for max_result_window -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Alex Arnaud <alex.arnaud@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact| |alex.arnaud@biblibre.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #12 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I don't have enough data to test this properly, but seems straight forward. Alex, maybe you could have another look? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #13 from Alex Arnaud <alex.arnaud@biblibre.com> --- (In reply to Katrin Fischer from comment #12)
I don't have enough data to test this properly, but seems straight forward. Alex, maybe you could have another look?
I already had many looks. I'd like Nick to valid/sign Julian's patches and i could pass QA. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com --- Comment #14 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Alex Arnaud from comment #13)
(In reply to Katrin Fischer from comment #12)
I don't have enough data to test this properly, but seems straight forward. Alex, maybe you could have another look?
I already had many looks. I'd like Nick to valid/sign Julian's patches and i could pass QA.
Sounds like a plan. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #15 from Julian Maurice <julian.maurice@biblibre.com> --- I wished Nick would review my patch to be sure that it doesn't go against his first patches. But I think this should not prevent the QA to be done. He can review the patches while pushing them ;) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #16 from Julian Maurice <julian.maurice@biblibre.com> --- (In reply to Katrin Fischer from comment #12)
I don't have enough data to test this properly, but seems straight forward. Alex, maybe you could have another look?
If you want, here is a tiny script to generate thousands of new biblio: https://nopaste.xyz/?6a713982d73e1ab2#44I4RKe2ynghhda8Gp48a9f3bQSXjWbPUoeawx... (You may want to change the title field for MARC21) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #73024|0 |1 is obsolete| | Attachment #74587|0 |1 is obsolete| | Attachment #75816|0 |1 is obsolete| | Attachment #76371|0 |1 is obsolete| | Attachment #76372|0 |1 is obsolete| | --- Comment #17 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 76788 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76788&action=edit Bug 19502: Limit pagination to first 10000 results when using ES This patch is to avoid hitting an error page. We should eventually make the max number returned configurable for ES. To test: 1 - Have Koha running ES with 10,000+ records 2 - Search for '*' 3 - Click 'Last' to view last page of results 4 - 'Cannot perform search' error 5 - Apply patch 6 - Search again 7 - View 'Last' page 8 - No error, you go to the last of 10000 9 - Note the warning above the pagination buttons Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #18 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 76789 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76789&action=edit Bug 19502: (follow-up) Pass parameters to avoid making templates depend on search engine https://bugs.koha-community.org/show_bug.cgi?id=19502 Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #19 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 76790 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76790&action=edit Bug 19502: Retrieve index.max_result_window from ES This avoid hardcoding '10000' in two different places and allow users to adjust this setting. Also, this patch fixes a bug when the search return less than 10000 results Test plan: 1. Do a search that returns 10000+ records. 2. Note the warning above the pagination buttons 3. Go to the last page, no error 4. Change the ES setting: curl -XPUT http://elasticsearch/koha_master_biblios/_settings -d \ '{"index": {"max_result_window": 20000}}' 5. Do another search that returns more than 10000 but less than 20000 6. Note that the warning does not show up 7. Go to the last page, still no error Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #20 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 76791 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76791&action=edit Bug 19502: Remove debug warn Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #21 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 76792 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76792&action=edit Bug 19502: Add POD for max_result_window Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Alex Arnaud <alex.arnaud@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #76788|0 |1 is obsolete| | --- Comment #22 from Alex Arnaud <alex.arnaud@biblibre.com> --- Created attachment 76794 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76794&action=edit Bug 19502: Limit pagination to first 10000 results when using ES This patch is to avoid hitting an error page. We should eventually make the max number returned configurable for ES. To test: 1 - Have Koha running ES with 10,000+ records 2 - Search for '*' 3 - Click 'Last' to view last page of results 4 - 'Cannot perform search' error 5 - Apply patch 6 - Search again 7 - View 'Last' page 8 - No error, you go to the last of 10000 9 - Note the warning above the pagination buttons Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Alex Arnaud <alex.arnaud@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #76789|0 |1 is obsolete| | --- Comment #23 from Alex Arnaud <alex.arnaud@biblibre.com> --- Created attachment 76795 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76795&action=edit Bug 19502: (follow-up) Pass parameters to avoid making templates depend on search engine https://bugs.koha-community.org/show_bug.cgi?id=19502 Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Alex Arnaud <alex.arnaud@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #76790|0 |1 is obsolete| | --- Comment #24 from Alex Arnaud <alex.arnaud@biblibre.com> --- Created attachment 76796 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76796&action=edit Bug 19502: Retrieve index.max_result_window from ES This avoid hardcoding '10000' in two different places and allow users to adjust this setting. Also, this patch fixes a bug when the search return less than 10000 results Test plan: 1. Do a search that returns 10000+ records. 2. Note the warning above the pagination buttons 3. Go to the last page, no error 4. Change the ES setting: curl -XPUT http://elasticsearch/koha_master_biblios/_settings -d \ '{"index": {"max_result_window": 20000}}' 5. Do another search that returns more than 10000 but less than 20000 6. Note that the warning does not show up 7. Go to the last page, still no error Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Alex Arnaud <alex.arnaud@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #76791|0 |1 is obsolete| | --- Comment #25 from Alex Arnaud <alex.arnaud@biblibre.com> --- Created attachment 76798 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76798&action=edit Bug 19502: Remove debug warn Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Alex Arnaud <alex.arnaud@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #76792|0 |1 is obsolete| | --- Comment #26 from Alex Arnaud <alex.arnaud@biblibre.com> --- Created attachment 76799 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=76799&action=edit Bug 19502: Add POD for max_result_window Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Alex Arnaud <alex.arnaud@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 --- Comment #27 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- yay! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master --- Comment #28 from Nick Clemens <nick@bywatersolutions.com> --- Awesome work all! Pushed to master for 18.11 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |martin.renvoize@ptfs-europe | |.com --- Comment #29 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Pushed to 18.05.x for 18.05.02 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=21405 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19502 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Stable |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org