https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #2 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200313 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200313&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #4 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200315 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200315&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #6 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200317 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200317&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pedro.amorim@openfifth.co.u | |k Status|Needs Signoff |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200312|0 |1 is obsolete| | Attachment #200313|0 |1 is obsolete| | Attachment #200314|0 |1 is obsolete| | Attachment #200315|0 |1 is obsolete| | Attachment #200316|0 |1 is obsolete| | Attachment #200317|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #9 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200606 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200606&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #11 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200608 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200608&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #13 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200610 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200610&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42643 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Strategic theme|--- |Performance Target Milestone|--- |26.11 Initiative type|--- |Epic -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|42643 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200605|0 |1 is obsolete| | Attachment #200606|0 |1 is obsolete| | Attachment #200607|0 |1 is obsolete| | Attachment #200608|0 |1 is obsolete| | Attachment #200609|0 |1 is obsolete| | Attachment #200610|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #15 from David Nind <david@davidnind.com> --- Created attachment 202139 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202139&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #17 from David Nind <david@davidnind.com> --- Created attachment 202141 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202141&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #19 from David Nind <david@davidnind.com> --- Created attachment 202143 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202143&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202138|0 |1 is obsolete| | Attachment #202139|0 |1 is obsolete| | Attachment #202140|0 |1 is obsolete| | Attachment #202141|0 |1 is obsolete| | Attachment #202142|0 |1 is obsolete| | Attachment #202143|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #22 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202230 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202230&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #24 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202232 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202232&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #26 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202234 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202234&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202229|0 |1 is obsolete| | Attachment #202230|0 |1 is obsolete| | Attachment #202231|0 |1 is obsolete| | Attachment #202232|0 |1 is obsolete| | Attachment #202233|0 |1 is obsolete| | Attachment #202234|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #28 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202481 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202481&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #30 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202483 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202483&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #32 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202485 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202485&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #34 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202487 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202487&action=edit Bug 42835: [DO NOT PUSH]: demo script for items index caching fix Standalone script demonstrating the caching fix in the previous commit. Wraps the real ES client to count calls, then calls _get_available_biblionumbers() and _get_item_facets() twice each against a live KTD Elasticsearch instance - first call should hit ES, second should be near-instant from cache. Run: perl demo_cache_fix.pl This commit is for local testing only and must not be pushed upstream. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Sponsorship status|--- |Sponsored Comma delimited| |Cuyahoga County Public list of Sponsors| |Library -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35052 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35052 [Bug 35052] OpacHiddenItemsHidesRecord system preference should be considered on index time instead -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #37 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 203718 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203718&action=edit Bug 42835: (QA follow-up) Make --items slice-aware for --processes _do_reindex_items() ignored %iterator_options and always reindexed the whole items table per forked process, instead of just its own slice. Test plan: 1) Run: perl misc/search_tools/rebuild_elasticsearch.pl --items --reset --processes 3 --verbose 2) Check the output: you should see 3 different "Processing slice" lines and 3 different item counts that add up to your total, not the same full count three times. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #39 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 203720 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203720&action=edit Bug 42835: (QA follow-up) Add test coverage for OPAC item visibility batching C4::Search::searchResults() gained a pre-scan that batches the OPAC item-visibility check into one DB call per page instead of one per item, but shipped with no corresponding test. Add coverage for a mixed visible/hidden batch, a batch with no items at all, and the per-item fallback path. The fallback is exercised by mocking MARC parsing to fail only during the pre-scan, forcing the old per-item lookup to run instead of the batched one. Test plan: 1) Run: prove t/db_dependent/Search.t Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #41 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 203722 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203722&action=edit Bug 42835: (QA follow-up) Document Koha::Item::_update_es_index This file documents its other private subs (_add_statistic, _status, _set_found_trigger) with a usage synopsis and description; _update_es_index only had a bare header. Fill it in to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #43 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Nice work Pedro.. apologies I didn't come back to this sooner.. I somehow missed the follow-ups had been posted :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202480|0 |1 is obsolete| | Attachment #202481|0 |1 is obsolete| | Attachment #202482|0 |1 is obsolete| | Attachment #202483|0 |1 is obsolete| | Attachment #202484|0 |1 is obsolete| | Attachment #202485|0 |1 is obsolete| | Attachment #202486|0 |1 is obsolete| | Attachment #202487|0 |1 is obsolete| | Attachment #203718|0 |1 is obsolete| | Attachment #203719|0 |1 is obsolete| | Attachment #203720|0 |1 is obsolete| | Attachment #203721|0 |1 is obsolete| | Attachment #203722|0 |1 is obsolete| | Attachment #203723|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #46 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205411 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205411&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #48 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205413 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205413&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #50 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205415 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205415&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #52 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205417 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205417&action=edit Bug 42835: [DO NOT PUSH]: demo script for items index caching fix Standalone script demonstrating the caching fix in the previous commit. Wraps the real ES client to count calls, then calls _get_available_biblionumbers() and _get_item_facets() twice each against a live KTD Elasticsearch instance - first call should hit ES, second should be near-instant from cache. Run: perl demo_cache_fix.pl This commit is for local testing only and must not be pushed upstream. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #54 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205419 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205419&action=edit Bug 42835: (QA follow-up) Don't skip biblio re-index on branch transfer A standalone holdingbranch change (transfer) was wrongly treated as safe to skip, same as a checkout. Now, when holdingbranch changes, the skip only applies if onloan or issues changed too. Test plan: 1) Run: prove t/db_dependent/Koha/Item.t Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #56 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205421 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205421&action=edit Bug 42835: (QA follow-up) Reuse pre-scanned MARC record in searchResults() The OPAC item-visibility pre-scan already parses every record on the page to collect itemnumbers, then the main loop parsed the same records again a few lines later. Stash the parsed record from the pre-scan and reuse it, instead of parsing twice. Test plan: 1) Run: prove t/db_dependent/Search.t Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #58 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205423 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205423&action=edit Bug 42835: [DO NOT PUSH] demo script for facets round-trip latency Benchmarks search_compat()'s two added costs on ~9k synthetic biblios (matching the _biblionumbers aggregation's own size cap): the extra terms aggregation on the biblios query, and the second round-trip to _get_item_facets. Cold-cache worst case adds ~25-55ms on top of a ~20-25ms baseline query, but the item-facets round-trip is cached, so repeated searches against the same result set pay far less. The script's verdict line checks the total round-trip against a configurable "still feels instant" budget ($ACCEPTABLE_TOTAL_MS, currently 100ms) rather than asserting a fixed conclusion -- every run so far has landed well under that budget, so no follow-up fix is warranted here. This commit is for local testing only and must not be pushed upstream. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #60 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205425 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205425&action=edit Bug 42835: (QA follow-up) Only treat a genuine holdingbranch value change as a transfer The previous fix for skipping the biblio re-index on branch transfers used exists $updated_columns{onloan} / {issues} as a proxy for "this is a checkout, not a transfer". That misses the case where a checkout also moves the item to a different branch than it currently has: onloan being dirty made is_branch_transfer false, so the branch-facet-relevant change was silently skipped along with the safe circ-only fields. Compare the actual holdingbranch value against the pre-store item instead of looking at which other columns happen to be dirty. A holdingbranch value that hasn't actually changed (AddIssue re-asserting the current branch alongside onloan) is still skippable; any real branch change now always forces a re-index, regardless of what else changed in the same store() call. Test plan: 1) Run: prove t/db_dependent/Koha/Item.t Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|Laura.escamilla@bywatersolu |aleisha@catalyst.net.nz |tions.com | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Luis Bataller <luis.bataller@xercode.es> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |luis.bataller@xercode.es -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #61 from David Cook <dcook@prosentient.com.au> --- I'll have to come take a look at this later. Sounds interesting though... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #63 from Luis Bataller <luis.bataller@xercode.es> --- Hi, Thanks for picking this up — indexing item data in Elasticsearch is something Koha has needed for a long time. We went through this same design question some months ago and evaluated both options: a separate item index, and items as `nested` objects inside the biblio document. We ended up choosing `nested`. We picked up the existing bug 8589 to offer that work to the community, but we never attached the patches there — we offered them and then let it drop, mostly on our side. The implementation is now in pre-production on an installation with roughly 1.5 million biblio records and 8 million items. A search filtered only by item attributes works fine with a separate index — that matches what bug 42835 currently implements, as far as I can tell. The difficulty we hit was with queries mixing item-level and biblio-level criteria, which is the common shape in both the OPAC and the staff interface. What follows is a summary of the analysis we carried out against that catalogue. --- Our reference case was the most frequent one: **library = b, title like "..."**. In that catalogue the largest library holds around half a million items, which resolves to somewhere between 140,000 and 320,000 distinct biblionumbers depending on the local items-per-record ratio — 10-20% of the whole catalogue in a single intermediate set. Two problems followed: 1. **Intermediate set size.** Passing that many biblionumbers as a `terms` filter exceeds `index.max_terms_count` (65,536 by default). The workarounds we considered were splitting the query into chunks, which breaks global scoring and relevance ordering, or a `terms lookup`, which introduces a synchronous write plus a refresh in the middle of a read path. 2. **Relevance pagination.** The selective criterion is the title, so the biblio query has to run first, but then there is no way to know how many of its hits will survive the item filter. Filling a page of 20 results becomes an iterative loop with an unpredictable number of round-trips, or requires fetching the full candidate set and re-sorting in application code, losing BM25 scoring. --- So my main question is: have you already worked out how general searches combining item and biblio criteria would be resolved, particularly these two points? If you have a solution, I would genuinely like to understand it — the separate-index approach has real advantages we would benefit from, particularly a much lower cost for full reindexing when an item field is added. If it turns out to be an open problem, we would be glad to share what we have. The patches are not attached to bug 8589 yet, but we can post them there promptly if there is interest, whether as something to contribute or just as a reference for comparison. We are also happy to run comparative tests against our dataset if that would help validate either design. Thanks again for working on this. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #65 from David Cook <dcook@prosentient.com.au> --- (In reply to Luis Bataller from comment #63)

Hi,

Thanks for picking this up — indexing item data in Elasticsearch is something Koha has needed for a long time.

We went through this same design question some months ago and evaluated both options: a separate item index, and items as `nested` objects inside the biblio document. We ended up choosing `nested`.

Great to meet you at the dev meeting last night, Luis. I agree with your analysis regarding having a separate item index vs having nested objects. To me, having a separate item index doesn't make sense, because it would be very inefficient at a large scale. That said, what have you observed with bib records with many (e.g. 200-1000 items) nested inside the JSON document in Elasticsearch? I am supportive of having nested items, because it allows for better searching, but I have only tried it with relatively small numbers of items embedded in the bib index. Actually with Elasticsearch I think that you can do includes/excludes with source filtering... so you could do a search against a large record but just retrieve the bib record when showing the search results... that could be very very useful. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #67 from David Cook <dcook@prosentient.com.au> --- @Pedro @Martin I'm short on time, so I was hoping you could give me a concise summary of how your patches work having a separate item index? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #69 from David Cook <dcook@prosentient.com.au> --- (In reply to Luis Bataller from comment #68)

Regarding the numbers, here is a table with real indexing time measurements:

items total(s) gen gen es_wall KB % only items (gen+es_wall) (total) (only items) ----- ------------- ------- ------------ ------- ------ ------------ 0 0.043 0.030 0.003 0.013 2.0 7.0 % 1 0.075 0.065 0.032 0.010 3.7 42.7 % 10 0.208 0.193 0.145 0.015 12.7 69.7 % 50 0.729 0.707 0.662 0.022 51.4 90.8 % 100 1.327 1.295 1.244 0.032 99.6 93.7 % 250 3.434 3.393 3.297 0.041 245.2 96.0 % 500 7.066 7.004 6.852 0.062 487.8 97.0 % 1000 13.864 13.718 13.267 0.146 1502.4 95.7 %

As you can see, indexing time grows a lot with the number of items; but if you look closely, most of the time (over 90% from 50 items on) is spent building the nested items. We are working on a solution to remove that hydration time, bringing it down to the time it takes to hydrate a single item. The idea is to take the already hydrated document from ES and replace only the item or items that have changed, so the cost of indexing a change would no longer depend on how many items the record has.

Any comments or suggestions you may have would be much appreciated.

I don't know if "hydration" is the right term to use there, since that refers to rendering data on a web page with Javascript. When you say "[time] spent building the nested items", are you referring to the Perl process of building the JSON document? Depend on the structure of the JSON document, doing partial updates might be the way to go for sure. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #71 from David Cook <dcook@prosentient.com.au> --- (In reply to Luis Bataller from comment #70)

Hi David. Yes, the spend building time is the time Perl needs to obtain all the data required to generate the JSON document.

Ok cool. Perhaps there's ways we could optimize that. Are you using Koha::Items to get the items? The ORM isn't optimised for performance, so it might be necessary to use a more targeted approach to get the time down. I've got some approaches/patterns for that. (I work on Perl systems which deal with a much higher volume of data than Koha so I'm happy to share lessons learned.) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@gmail.com See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=38181 --- Comment #73 from Jonathan Druart <jonathan.druart@gmail.com> --- From https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=207793 Bug 38181: Look up item statuses once per record in buildKohaItemsNamespace On a record with 1000 items (koha-testing-docker) the routine goes from 0.85s to 0.09s, and an OPAC searchResults call for that record from 1.2s to 0.45s. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #75 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- I am working here on testing both methods to compare, based on the description of issues from Luis I think that subdocuments may be a better path, it was what I had looked into when researching this previously. A quick google provides a few pro cons for the two: parent-child is faster for indexing, but slower for searching subdocuments can be used in sort - not in parent-child parent-child is recommended for write heavy operations due to indexing subdocuments requiring the whole object be reindexed Overall, I think subdocuments perform as we want and we will need to focus on indexing speed. I am currently building out a large dataset for testing and will update as I progress -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #77 from David Cook <dcook@prosentient.com.au> --- At the moment, I'm in the process of indexing tens/hundreds of millions of rows of non-Koha data into Elasticsearch, but it's a fairly straight forward de-normalised JSON document for each row, so not the same use case at all, but it'll be interesting from a performance perspective... -- Looking again at parent-join it says: "The only case where the join field makes sense is if your data contains a one-to-many relationship where one entity significantly outnumbers the other entity. An example of such case is a use case with products and offers for these products. In the case that offers significantly outnumbers the number of products then it makes sense to model the product as parent document and the offer as child document." Hmm tricky... that does seem to describe the biblio to item model... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #79 from Luis Bataller <luis.bataller@xercode.es> --- (In reply to David Cook from comment #78)

Question... what do these 2 options have in common and what do they not have in common in terms of code?

Would it be possible to incorporate both options into Koha on an experimental basis?

--

On other bug reports and on Mattermost, I've been advocating for a design approach that embraces modularity more, so that we can more easily swap out modules.

As a community, we probably don't have the resources to support both, but if we design an internal API that allows us to use different modules, it will allow for easier innovations in the short-erm, which should also help with decision making for the longer term.

We could also end up choosing one as the community default, while a support company could choose the other option, and support it themselves.

I think there's some interesting possibilities here...

I think it's complex, but feasible. We could create two search subclasses for Elasticsearch: Koha::SearchEngine::ElasticsearchNested and Koha::SearchEngine::ElasticsearchParentChild. But I think the main challenge lies in the definitions (mappings, facets, sort fields, etc.) and in handling item updates that shouldn't trigger a reindex of the parent document. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #81 from Luis Bataller <luis.bataller@xercode.es> --- Hi Martin, Thank you for taking the time to check this. A note about something in the development that we hadn't explained. When we started, we decided not to change the meaning of the "available" field: "all items except lost and checked-out ones", which we understood as availability from the staff point of view. For patron availability we created a new field, items.available_for_patron. Its default conditions exclude lost, withdrawn and damaged items, items in transit and items held for a hold. Plugins can override these conditions. We added this because availability policies vary between organizations. About the "cross-item" result in Q9: this branch doesn't remove the flat item fields from the record document yet (that is the next phase of the development). As a result, the legacy available limit falls back to the flat. The same happens with itype/ccode limits. The UI doesn't emit those forms: facets emit items.itype:BK and the advanced search emits mc-itype:BK. To force the nested, use items.available:true or items.available_for_patron:true -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #205411|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #205413|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #205415|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #205417|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #205419|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #205421|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #205423|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #205425|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #82 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Hi all, Thank you again, Luis, and thanks to everyone who has contributed to the discussion here. We have now benchmarked the two designs against each other on the same dataset (the scripts are in attachment 208519), and we have discussed the result internally, including with Pedro, who wrote the separate items index patches on this bug. Our conclusion is that nested items is the right way forward for Koha, and that we should pick that one design rather than try to maintain two. In short: * On a 250k biblio / 1.26M item catalogue where one library holds 40% of the items, queries combining a title with an item-level limit (library, availability, item type + collection) took between 0.8 and 12.8 seconds cold on the separate index, against roughly 100 to 200 ms with nested items. The separate index has to fetch every matching biblionumber from the items index and pass them back as an ids filter, so its cost grows with the item-side set rather than with the query; a 39-hit title search limited to the large library took 7.3 seconds. Hit counts matched the independent oracle on both sides for same-item combinations. * Nested items gives same-item semantics, sorting and relevance-ordered pagination natively, without the intermediate set that Luis described in comment 63. * The write-side advantage of a separate index (no biblio reindex on a checkout) turned out to be small next to the read-side cost: on nested, a checkout-shaped change cost 27-39 ms at store time, plus a background reindex of 50-370 ms for records with 1 to 900 items. What this means for this bug: 1. We have obsoleted the patches attached here (the separate items index series). Pedro and I have talked this through and he is happy to proceed this way. Thank you Pedro, for framing the problem so well; several of the ideas carry over (the availability definition, batching the OPAC visibility checks, the reactive reindex triggers). 2. We would like to repurpose this bug (42835) as the home of the first piece of the nested items work: "index items as nested objects in the biblio document, together with the reindexing that keeps them correct". Luis, we would love for you to attach that first part of your branch here. 3. The branch is substantial (38 commits, about 5,600 lines), and it will be much easier for the community to review, test and QA if it is split across a small number of dependent bugs. We have done a QA review of it and will post that, together with a suggested split and some notes on the submission process, in a follow-up comment. The short version: the design is sound and the work is impressive. There are some changes needed, but all of them are fixable, and we would like to help get this over the line. David, thank you for the suggestion of supporting both. Given the limited QA resources we all have, we think picking one is the better path for now, but the modularity idea is a good one to keep in mind for the longer term. Thanks all. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #84 from Luis Bataller <luis.bataller@xercode.es> --- Hi all, Thanks a lot to everyone for the time and work you've put into this: the benchmark, the review and the discussion. Pedro, thanks for being so generous about your patches. If that works for you, our plan is to start with the three blockers, fixing them on the current branch first. For the Zebra one, we'll do both of the suggested fixes, the second is usefull in case an installation switches from Elasticsearch to Zebra and has saved searches or URLs. We'll also include two of the important points, as they're simple to fix and affect data integrity and visibility: failing closed when a plugin dies, and wrapping the index_records calls and Library::store in try/catch. Martin, once that's done we'll put a split proposal based on your suggestions. Since you have much more experience with this, we'd really appreciate it if you could review it before we file the bugs. Thanks -- You are receiving this mail because: You are watching all bug changes.