[Bug 22246] New: Elasticsearch indexing needs a maximum length for __sort fields
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Bug ID: 22246 Summary: Elasticsearch indexing needs a maximum length for __sort fields Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: ere.maijala@helsinki.fi Target Milestone: --- Currently there's no limit to the length of *__sort fields. If such field is created from item information or something else that can be repeated a lot, the field may become too long. Such field is e.g. local-classification, which is by default created from 952o. If there are hundreds or thousands of items, the field may become longer than what Elasticsearch accepts. Also consider making it a non-sortable field by default. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |ere.maijala@helsinki.fi |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 --- Comment #1 from Ere Maijala <ere.maijala@helsinki.fi> --- Created attachment 84607 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=84607&action=edit Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Also fixes mapping for llevel while at it. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #84607|0 |1 is obsolete| | --- Comment #2 from Ere Maijala <ere.maijala@helsinki.fi> --- Created attachment 84608 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=84608&action=edit Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - low |P3 Severity|enhancement |normal -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #84608|0 |1 is obsolete| | --- Comment #3 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 84626 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=84626&action=edit Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #84626|0 |1 is obsolete| | --- Comment #4 from Josef Moravec <josef.moravec@gmail.com> --- Created attachment 85863 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=85863&action=edit Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA CC| |josef.moravec@gmail.com QA Contact| |josef.moravec@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Patch doesn't apply CC| |nick@bywatersolutions.com --- Comment #5 from Nick Clemens <nick@bywatersolutions.com> --- Please rebase -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #85863|0 |1 is obsolete| | --- Comment #6 from Ere Maijala <ere.maijala@helsinki.fi> --- Created attachment 86050 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=86050&action=edit Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 --- Comment #7 from Ere Maijala <ere.maijala@helsinki.fi> --- Rebased. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master --- Comment #8 from Nick Clemens <nick@bywatersolutions.com> --- Awesome work all! Pushed to master for 19.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to Master |Pushed to Stable CC| |martin.renvoize@ptfs-europe | |.com --- Comment #9 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Pushed to 18.11.x for 18.11.04 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com --- Comment #10 from Lucas Gass <lucas@bywatersolutions.com> --- not backporting to 18.05.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22246 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Elasticsearch indexing |Elasticsearch indexing |needs a maximum length for |needs a maximum length for |__sort fields |`__sort` fields -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org