[Bug 24428] New: Elastic search index does not contain consistent item information
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 Bug ID: 24428 Summary: Elastic search index does not contain consistent item information Change sponsored?: --- Product: Koha Version: 18.11 Hardware: PC OS: Linux Status: NEW Severity: critical Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: david.holoshka@ub.lu.se Probably when catmandu was removed from the indexing process the item information organization was lost. For example whenever there is more than a one copy of an item any item's missing information is just omitted from the es array - no place holder is used. Previously the 3rd item's information was always found in the 3 position in the items fields now this can no longer be assumed because if item 1 or 2 is missing data for this field then the 3rd item's information is placed in either position 1 or 2. This means that es can not be used to extract item information extra database look ups are required. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 --- Comment #1 from david holoshka <david.holoshka@ub.lu.se> --- also the local-classification for a single item is stored using multiple array positions in elastic search instead of a single array element. This breaks with the previous structure of storage. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ere.maijala@helsinki.fi, | |katrin.fischer@bsz-bw.de, | |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 --- Comment #2 from Ere Maijala <ere.maijala@helsinki.fi> --- I don't think you can ever rely on the field order being correct. In Elasticsearch the documents are JSON objects, and a JSON object is by definition "an unordered set of name/value pairs". The way around this would be to either include item information in MARC or index each item as a subdocument. This should probably be optional. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 --- Comment #3 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- This is a problem we currently have with Zebra, repeated 952 fields are indexed as without keeping the information separate per item. So you can't search for something like 'item of itemtype X in library B'. The search will also bring up all results where one item is in library B and antoher item has itemtype X. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 --- Comment #4 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Saying: it would be really great to fix this with Elastic. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|18.11 |master CC| |jonathan.druart@bugs.koha-c | |ommunity.org Severity|critical |enhancement --- Comment #5 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I would consider this more like an enhancement request than a critical bug. We will need to create a new index for items, which implies a quite big work. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 --- Comment #6 from Ere Maijala <ere.maijala@helsinki.fi> --- Workaround, or a solution depending on how you view this, is to set the ElasticsearchMARCFormat pref to "Searchable array" and reindex everything. Then the item information would be available in the 952 field. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Elastic search index does |Elasticsearch index does |not contain consistent item |not contain consistent item |information |information -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 --- Comment #7 from david holoshka <david.holoshka@ub.lu.se> --- Hi, I don't really follow how breaking functionality then say that restoring the functionality is a enhancement. We have several modifications to koha that depended on finding organized item information in the es index. It wase previously structured so that index 0, for example, for all item info always belonged to the same item. I thought the community was moving towards es I so I used this fast searching ability to build new functionality. It is a waste of time to lookup something in an index that contains all the info then look up the item again in a database or in koha's case look up biblio info again in the biblio tables. If the item information is organized in es we can make use of es fast search capabilities to improve koha's performance for a lot more scenarios eg subscriptions. We are working to optimize the koha search here, but it is going to take awhile. Best Regards, David -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 --- Comment #8 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to david holoshka from comment #7)
We have several modifications to koha that depended on finding organized item information in the es index.
Hi David, Are these modifications available to the community in bugzilla or github so that we can see how they work? We are striving to improve the Elasticsearch functionality and most of the work has been put into efforts to improve the basic search functionality. While we have talked about using ES as a data store, we have not focused on this area yet, which is why I think we would call this an enhancement. Classifying it this way doesn't mean we don't think it is important or necessary, just that it doesn't break the features built in to Koha at this time. I would try switching the system preference ElasticsearchMARCFormat as Ere mentions, as this indexes the full marc record as a searchable array and may help with your issue. -Nick -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 --- Comment #9 from david holoshka <david.holoshka@ub.lu.se> --- Hi Nick, Does this mean you are not against making the es index backward compatible ? I also noticed that my code broke because the marc record in es is now in some encoded format (or in some cases marcxml) instead of json. I will work on a patch for the indexer to make it backward compatible. We will try to add es 6.5 support with aliasing to make index rebuilds possible during working hours. We are setting the default for marc to marcxml as it is easier to debug. I guess the base64 format uses a little less space. Thanks for considering this problem. Best Regards, David -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 --- Comment #10 from Ere Maijala <ere.maijala@helsinki.fi> --- David, please see comment #6. There's no need to to additional work. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24428 --- Comment #11 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I slightly disagree with Nick in that I think item searching is part of the basic search we are aiming at right now. The itemtype, location and collection searches and facetting on item values are prominent features in our OPAC and basically they never worked correctly with Zebra. You can argue that this is not a regression compared to how Zebra worked (it's just the same bug as before...), but you could also say we should work to get rid if our old problems with a new implementation :) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org