https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #2 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200313 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200313&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #4 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200315 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200315&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #6 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200317 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200317&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pedro.amorim@openfifth.co.u | |k Status|Needs Signoff |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200312|0 |1 is obsolete| | Attachment #200313|0 |1 is obsolete| | Attachment #200314|0 |1 is obsolete| | Attachment #200315|0 |1 is obsolete| | Attachment #200316|0 |1 is obsolete| | Attachment #200317|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #9 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200606 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200606&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #11 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200608 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200608&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #13 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200610 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200610&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42643 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Strategic theme|--- |Performance Target Milestone|--- |26.11 Initiative type|--- |Epic -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|42643 | Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #200605|0 |1 is obsolete| | Attachment #200606|0 |1 is obsolete| | Attachment #200607|0 |1 is obsolete| | Attachment #200608|0 |1 is obsolete| | Attachment #200609|0 |1 is obsolete| | Attachment #200610|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #15 from David Nind <david@davidnind.com> --- Created attachment 202139 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202139&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #17 from David Nind <david@davidnind.com> --- Created attachment 202141 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202141&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #19 from David Nind <david@davidnind.com> --- Created attachment 202143 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202143&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202138|0 |1 is obsolete| | Attachment #202139|0 |1 is obsolete| | Attachment #202140|0 |1 is obsolete| | Attachment #202141|0 |1 is obsolete| | Attachment #202142|0 |1 is obsolete| | Attachment #202143|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #22 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202230 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202230&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #24 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202232 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202232&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #26 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202234 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202234&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202229|0 |1 is obsolete| | Attachment #202230|0 |1 is obsolete| | Attachment #202231|0 |1 is obsolete| | Attachment #202232|0 |1 is obsolete| | Attachment #202233|0 |1 is obsolete| | Attachment #202234|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #28 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202481 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202481&action=edit Bug 42835: Koha::Item: Skip biblio re-index for circulation-only field changes When only fields in %ITEM_CIRC_FIELDS are modified (onloan, issues, renewals, holdingbranch, datelastborrowed, datelastseen, localuse, reserves, timestamp), skip the biblio MARC re-index — the items index is authoritative for this data. notforloan, damaged, itemlost and withdrawn are excluded from the skip set: they affect the available field in the biblios index and must trigger a re-index to keep availability filtering correct. The skip is further gated on _items_index_ready(): the biblio re-index is only skipped when the items index is sufficiently populated (>=95% of DB items indexed). This prevents stale availability data on fresh installs where the items index may contain only a handful of documents. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #30 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202483 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202483&action=edit Bug 42835: Elasticsearch: Route availability and facets through items index Queries the items index for availability filtering and item-level facets (itype, location, ccode, homebranch, holdingbranch). Uses cardinality sub-aggregations for per-biblio facet counts rather than per-item counts. Adds biblionumber as an explicit integer field in the biblios index mapping and stamps it on biblio documents so item facet queries can be scoped to the current search result set via a _biblionumbers aggregation. Fixes a bug where _apply_available_filter assumed available:true was always at must[0]; now scans all must elements. Fixes a bug where biblio_count{value} was tested for hashref truthiness rather than key existence, always preferring it over doc_count. Fixes the biblios index available field to correctly treat notforloan, damaged, and withdrawn items as unavailable — previously only onloan and itemlost were checked. Covers both item-level notforloan and items whose item type has notforloan set. Updates existing tests in Elasticsearch.t that were asserting the previous wrong behaviour. Items index routing is unconditional here; the readiness gate that prevents empty-index regressions on upgrade is added in the next commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #32 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202485 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202485&action=edit Bug 42835: Elasticsearch: Gate items index features behind readiness check Adds _items_index_ready() which checks whether the items index is fully populated before enabling availability filtering and facet aggregation. The index is considered ready only when it contains at least 95% of the items in the database — this prevents the gate opening after a single circulation event has indexed one item before rebuild_elasticsearch.pl --items has been run. The result is cached: 5 minutes when ready, 60 seconds when not, so a freshly populated index is detected quickly without a restart. TEST PLAN (Do not apply patches yet) 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you get 40 results. 9 of these bibs have all their items unavailable and should NOT be showing. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. 9) Let's pick a 2nd bib from that same search ('music' + only available): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 10) Edit that single item, put it 'Not for loan' 11) Refresh the search. Notice the bib is always showing, even though all its items are unavailable (This is incorrect): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available APPLY PATCHES 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Do a search for 'music', you should get 42 results: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music 3) Click the 'Limit to records with available items' facet link. Notice you now get 31 results. The previous 9 with all their items unavailable do not show anymore. 4) Find a biblio with a single item (the top result of the search): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=437 5) Check-out that item (39999000019186) to a patron 6) Repeat the same search with 'Limit to records with available items'. Notice the biblio above is no longer showing (this is still correct). 7) Check the background jobs: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 8) Confirm at least 1 'Update Elasticsearch index' entry exists. This is because a full bib reindex is triggered for anytime a check-out happens. This is still the case because a full reindex (to create the items index) has not occurred yet. Let's do that 9) Run: $ perl misc/search_tools/rebuild_elasticsearch.pl -i -r -v 10) Do another check-out for a 2nd bib with a single item e.g. (39999000016710): http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=380 11) Check the background jobs again: http://localhost:8081/cgi-bin/koha/admin/background_jobs.pl 12) Notice a new full bib record reindex job was not triggered, because this was updated directly on the items index synchronously. Confirm that bib record no longer shows on the 'only available' search: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available 13) Pick a third bib from the same search http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=374 14) Edit the single item and put it 'not for loan'. 15) Refresh the search. Notice the bib is not showing anymore, because all its items are unavailable (This is now correct): http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=&q=music&weight_search=1&sort_by=relevance&limit=available Benefits: 1. notforloan/damaged/withdrawn availability fix 2. Skip biblio re-index on checkout — performance improvement, visible under load 3. Availability filtering via items index — real-time post-checkout 4. Safe on upgrade — falls back to the biblios index until a rebuild with --items is run, no hard requirement to rebuild before the system works correctly. Run tests: prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t prove t/db_dependent/Koha/Item.t prove t/db_dependent/Search.t prove t/db_dependent/Koha/SearchEngine/Search.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 --- Comment #34 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202487 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202487&action=edit Bug 42835: [DO NOT PUSH]: demo script for items index caching fix Standalone script demonstrating the caching fix in the previous commit. Wraps the real ES client to count calls, then calls _get_available_biblionumbers() and _get_item_facets() twice each against a live KTD Elasticsearch instance - first call should hit ES, second should be near-instant from cache. Run: perl demo_cache_fix.pl This commit is for local testing only and must not be pushed upstream. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Sponsorship status|--- |Sponsored Comma delimited| |Cuyahoga County Public list of Sponsors| |Library -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35052 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35052 [Bug 35052] OpacHiddenItemsHidesRecord system preference should be considered on index time instead -- You are receiving this mail because: You are watching all bug changes.