https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #2 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200313 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200313&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #4 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200315 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200315&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #6 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200317 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200317&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pedro.amorim@openfifth.co.u | |k Status|Needs Signoff |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200312|0 |1 is obsolete| | Attachment #200313|0 |1 is obsolete| | Attachment #200314|0 |1 is obsolete| | Attachment #200315|0 |1 is obsolete| | Attachment #200316|0 |1 is obsolete| | Attachment #200317|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #9 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200606 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200606&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #11 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200608 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200608&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #13 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200610 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200610&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42643 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Strategic theme|--- |Performance Target Milestone|--- |26.11 Initiative type|--- |Epic -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|42643 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200605|0 |1 is obsolete| | Attachment #200606|0 |1 is obsolete| | Attachment #200607|0 |1 is obsolete| | Attachment #200608|0 |1 is obsolete| | Attachment #200609|0 |1 is obsolete| | Attachment #200610|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #15 from David Nind <david@davidnind.com> --- Created attachment 202139 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202139&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #17 from David Nind <david@davidnind.com> --- Created attachment 202141 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202141&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #19 from David Nind <david@davidnind.com> --- Created attachment 202143 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202143&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202138|0 |1 is obsolete| | Attachment #202139|0 |1 is obsolete| | Attachment #202140|0 |1 is obsolete| | Attachment #202141|0 |1 is obsolete| | Attachment #202142|0 |1 is obsolete| | Attachment #202143|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #22 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202230 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202230&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #24 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202232 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202232&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #26 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202234 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202234&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202229|0 |1 is obsolete| | Attachment #202230|0 |1 is obsolete| | Attachment #202231|0 |1 is obsolete| | Attachment #202232|0 |1 is obsolete| | Attachment #202233|0 |1 is obsolete| | Attachment #202234|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #28 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202481 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202481&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #30 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202483 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202483&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #32 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202485 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202485&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #34 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202487 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202487&action=edit Bug 42835: [DO NOT PUSH]: demo script for items index caching fix Standalone script demonstrating the caching fix in the previous commit. Wraps the real ES client to count calls, then calls _get_available_biblionumbers() and _get_item_facets() twice each against a live KTD Elasticsearch instance - first call should hit ES, second should be near-instant from cache. Run: perl demo_cache_fix.pl This commit is for local testing only and must not be pushed upstream. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Sponsorship status|--- |Sponsored Comma delimited| |Cuyahoga County Public list of Sponsors| |Library -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35052 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35052 [Bug 35052] OpacHiddenItemsHidesRecord system preference should be considered on index time instead -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #37 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 203718 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203718&action=edit Bug 42835: (QA follow-up) Make --items slice-aware for --processes _do_reindex_items() ignored %iterator_options and always reindexed the whole items table per forked process, instead of just its own slice. Test plan: 1) Run: perl misc/search_tools/rebuild_elasticsearch.pl --items --reset --processes 3 --verbose 2) Check the output: you should see 3 different "Processing slice" lines and 3 different item counts that add up to your total, not the same full count three times. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #39 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 203720 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203720&action=edit Bug 42835: (QA follow-up) Add test coverage for OPAC item visibility batching C4::Search::searchResults() gained a pre-scan that batches the OPAC item-visibility check into one DB call per page instead of one per item, but shipped with no corresponding test. Add coverage for a mixed visible/hidden batch, a batch with no items at all, and the per-item fallback path. The fallback is exercised by mocking MARC parsing to fail only during the pre-scan, forcing the old per-item lookup to run instead of the batched one. Test plan: 1) Run: prove t/db_dependent/Search.t Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #41 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 203722 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=203722&action=edit Bug 42835: (QA follow-up) Document Koha::Item::_update_es_index This file documents its other private subs (_add_statistic, _status, _set_found_trigger) with a usage synopsis and description; _update_es_index only had a bare header. Fill it in to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #43 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Nice work Pedro.. apologies I didn't come back to this sooner.. I somehow missed the follow-ups had been posted :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202480|0 |1 is obsolete| | Attachment #202481|0 |1 is obsolete| | Attachment #202482|0 |1 is obsolete| | Attachment #202483|0 |1 is obsolete| | Attachment #202484|0 |1 is obsolete| | Attachment #202485|0 |1 is obsolete| | Attachment #202486|0 |1 is obsolete| | Attachment #202487|0 |1 is obsolete| | Attachment #203718|0 |1 is obsolete| | Attachment #203719|0 |1 is obsolete| | Attachment #203720|0 |1 is obsolete| | Attachment #203721|0 |1 is obsolete| | Attachment #203722|0 |1 is obsolete| | Attachment #203723|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #46 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205411 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205411&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #48 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205413 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205413&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #50 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205415 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205415&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #52 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205417 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205417&action=edit Bug 42835: [DO NOT PUSH]: demo script for items index caching fix Standalone script demonstrating the caching fix in the previous commit. Wraps the real ES client to count calls, then calls _get_available_biblionumbers() and _get_item_facets() twice each against a live KTD Elasticsearch instance - first call should hit ES, second should be near-instant from cache. Run: perl demo_cache_fix.pl This commit is for local testing only and must not be pushed upstream. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #54 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205419 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205419&action=edit Bug 42835: (QA follow-up) Don't skip biblio re-index on branch transfer A standalone holdingbranch change (transfer) was wrongly treated as safe to skip, same as a checkout. Now, when holdingbranch changes, the skip only applies if onloan or issues changed too. Test plan: 1) Run: prove t/db_dependent/Koha/Item.t Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #56 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205421 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205421&action=edit Bug 42835: (QA follow-up) Reuse pre-scanned MARC record in searchResults() The OPAC item-visibility pre-scan already parses every record on the page to collect itemnumbers, then the main loop parsed the same records again a few lines later. Stash the parsed record from the pre-scan and reuse it, instead of parsing twice. Test plan: 1) Run: prove t/db_dependent/Search.t Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #58 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205423 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205423&action=edit Bug 42835: [DO NOT PUSH] demo script for facets round-trip latency Benchmarks search_compat()'s two added costs on ~9k synthetic biblios (matching the _biblionumbers aggregation's own size cap): the extra terms aggregation on the biblios query, and the second round-trip to _get_item_facets. Cold-cache worst case adds ~25-55ms on top of a ~20-25ms baseline query, but the item-facets round-trip is cached, so repeated searches against the same result set pay far less. The script's verdict line checks the total round-trip against a configurable "still feels instant" budget ($ACCEPTABLE_TOTAL_MS, currently 100ms) rather than asserting a fixed conclusion -- every run so far has landed well under that budget, so no follow-up fix is warranted here. This commit is for local testing only and must not be pushed upstream. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #60 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205425 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205425&action=edit Bug 42835: (QA follow-up) Only treat a genuine holdingbranch value change as a transfer The previous fix for skipping the biblio re-index on branch transfers used exists $updated_columns{onloan} / {issues} as a proxy for "this is a checkout, not a transfer". That misses the case where a checkout also moves the item to a different branch than it currently has: onloan being dirty made is_branch_transfer false, so the branch-facet-relevant change was silently skipped along with the safe circ-only fields. Compare the actual holdingbranch value against the pre-store item instead of looking at which other columns happen to be dirty. A holdingbranch value that hasn't actually changed (AddIssue re-asserting the current branch alongside onloan) is still skippable; any real branch change now always forces a re-index, regardless of what else changed in the same store() call. Test plan: 1) Run: prove t/db_dependent/Koha/Item.t Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|Laura.escamilla@bywatersolu |aleisha@catalyst.net.nz |tions.com | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Luis Bataller <luis.bataller@xercode.es> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |luis.bataller@xercode.es -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #61 from David Cook <dcook@prosentient.com.au> --- I'll have to come take a look at this later. Sounds interesting though... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #63 from Luis Bataller <luis.bataller@xercode.es> --- Hi, Thanks for picking this up — indexing item data in Elasticsearch is something Koha has needed for a long time. We went through this same design question some months ago and evaluated both options: a separate item index, and items as `nested` objects inside the biblio document. We ended up choosing `nested`. We picked up the existing bug 8589 to offer that work to the community, but we never attached the patches there — we offered them and then let it drop, mostly on our side. The implementation is now in pre-production on an installation with roughly 1.5 million biblio records and 8 million items. A search filtered only by item attributes works fine with a separate index — that matches what bug 42835 currently implements, as far as I can tell. The difficulty we hit was with queries mixing item-level and biblio-level criteria, which is the common shape in both the OPAC and the staff interface. What follows is a summary of the analysis we carried out against that catalogue. --- Our reference case was the most frequent one: **library = b, title like "..."**. In that catalogue the largest library holds around half a million items, which resolves to somewhere between 140,000 and 320,000 distinct biblionumbers depending on the local items-per-record ratio — 10-20% of the whole catalogue in a single intermediate set. Two problems followed: 1. **Intermediate set size.** Passing that many biblionumbers as a `terms` filter exceeds `index.max_terms_count` (65,536 by default). The workarounds we considered were splitting the query into chunks, which breaks global scoring and relevance ordering, or a `terms lookup`, which introduces a synchronous write plus a refresh in the middle of a read path. 2. **Relevance pagination.** The selective criterion is the title, so the biblio query has to run first, but then there is no way to know how many of its hits will survive the item filter. Filling a page of 20 results becomes an iterative loop with an unpredictable number of round-trips, or requires fetching the full candidate set and re-sorting in application code, losing BM25 scoring. --- So my main question is: have you already worked out how general searches combining item and biblio criteria would be resolved, particularly these two points? If you have a solution, I would genuinely like to understand it — the separate-index approach has real advantages we would benefit from, particularly a much lower cost for full reindexing when an item field is added. If it turns out to be an open problem, we would be glad to share what we have. The patches are not attached to bug 8589 yet, but we can post them there promptly if there is interest, whether as something to contribute or just as a reference for comparison. We are also happy to run comparative tests against our dataset if that would help validate either design. Thanks again for working on this. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #65 from David Cook <dcook@prosentient.com.au> --- (In reply to Luis Bataller from comment #63)

Hi,

Thanks for picking this up — indexing item data in Elasticsearch is something Koha has needed for a long time.

We went through this same design question some months ago and evaluated both options: a separate item index, and items as `nested` objects inside the biblio document. We ended up choosing `nested`.

Great to meet you at the dev meeting last night, Luis. I agree with your analysis regarding having a separate item index vs having nested objects. To me, having a separate item index doesn't make sense, because it would be very inefficient at a large scale. That said, what have you observed with bib records with many (e.g. 200-1000 items) nested inside the JSON document in Elasticsearch? I am supportive of having nested items, because it allows for better searching, but I have only tried it with relatively small numbers of items embedded in the bib index. Actually with Elasticsearch I think that you can do includes/excludes with source filtering... so you could do a search against a large record but just retrieve the bib record when showing the search results... that could be very very useful. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #67 from David Cook <dcook@prosentient.com.au> --- @Pedro @Martin I'm short on time, so I was hoping you could give me a concise summary of how your patches work having a separate item index? -- You are receiving this mail because: You are watching all bug changes.