https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #74 from Luis Bataller <luis.bataller@xercode.es> --- Hi all! Quick update on the nested item index (PR #1 on xercode/koha-item-index: https://github.com/xercode/koha-item-index/pull/1). Since our last post we've been working on indexing speed, especially for records with lots of items, and we've merged two improvements into the branch: 1. Look up the item hook plugins once per batch. Building each item called get_enabled_plugins twice per item. Now we call it once per indexing batch and reuse the result. 2. Store each item's MARC field in the nested items (items.marc). The item fields are no longer embedded in the record's MARC in the main document; instead, each item's MARC field is stored in its nested item. For records with many items, the record fits in ISO2709 again instead of falling back to MARCXML, so their documents are about 40% smaller and decoding them is much faster. Here's what it takes to reindex a single record from start to finish: Items per record Before After improvement 1 After improvement 2 1 0.07 s 0.06 s 0.07 s 10 0.14 s 0.09 s 0.07 s 100 0.88 s 0.31 s 0.14 s 1,000 9.1 s 2.9 s 0.73 s 5,000 47.0 s 15.6 s 3.96 s For a full reindex of our test catalogue (MARC21, 2,068 records and 25,631 items, one process): 255 s → 94 s → 40 s. Under a burst of circulation (with the indexer's 10-second pause between batches disabled), the median delay before a change shows up in the index dropped from 34 s to about 1 s. What's next: dropping the flat item fields Right now every item field mapped to a search field (952/995) is indexed twice: once as a flat field on the record, which merges all the items together, and once in the nested items. We already have a branch that removes the flat copies. With it the index is about 26% smaller, big records index faster, and items hidden in the OPAC can no longer make a record show up in search results. The catch is that it changes how search behaves: - keyword searches no longer match item fields such as public notes - custom mappings on 952/995 have to be searched with the items. prefix - facets on item fields other than the default ones need extra work We have workarounds for most of these, but they come at a cost: slightly bigger documents, more complex queries and somewhat slower keyword searches. And a keyword search mixing record and item terms (a word from the title plus one from an item note) would still not match. So we don't want to merge it before talking it over with you. How would you approach this? Should we carry on in this thread, or would a new one be better for this discussion? Thanks -- You are receiving this mail because: You are watching all bug changes.