[Bug 43151] New: ES should apply search filters on each item discretely
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Bug ID: 43151 Summary: ES should apply search filters on each item discretely Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: andrew@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org Depends on: 42835 Target Milestone: 26.11 When filtering search results using item record values, Koha checks for each of those values at the biblio level, rather than at the item level. This leads to undesirable search results. For example, assume you have a biblio with two items: - item 1 is itype BOOK and collection FICTION - item 2 is itype AUDIO and collection NONFICTION Currently, if a user performs a search limited to itype=BOOK and collection=NONFICTION, this biblio will be returned because it has at least one BOOK item and at least one NONFICTION item. But the user expectation is that the search will return biblios where at least one item that is both a BOOK and NONFICTION. Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 [Bug 42835] ElasticSearch should have an items index -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 --- Comment #1 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202512 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202512&action=edit Bug 43151: Require a single item to satisfy combined item-level search filters Combined item-level limits (item type, collection, location, home/holding library, availability) were checked independently against each bib's flattened, cross-item field arrays, so a bib could match a filter combination no single item actually satisfied (e.g. itype:BOOK + ccode:FICTION matching a bib whose only book is non-fiction and whose only fiction item is a CD). QueryBuilder now extracts these limits into constraint groups before they're folded into the query string, and Search resolves them as one combined query against the items index, requiring a single item to satisfy all of them together. Falls back to the existing (imperfect) query-string behaviour if the items index isn't ready. Note: this does not fix the item-level facet counts shown alongside these results, which are still built from the same flattened biblio-level arrays and can still list values that don't reflect the active combined filter. That's a separate, not yet addressed, follow-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 --- Comment #2 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202513 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202513&action=edit Bug 43151: Scope item-level facet counts to the active combined filter Facet counts (itype, location, ccode, homebranch, holdingbranch) scanned every item on each matching bib, ignoring the active combined filter - filtering to itype:BOOK + ccode:FICTION still listed unrelated values like Music in the itype facet. _get_item_facets now runs one query per facet field, each excluding that field's own constraint but keeping the others, so facets stay accurate without hiding valid alternative values (standard multi-select faceting). Test plan: 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Catalogue a record titled (MARC21: 245$a) e.g. 'ZZZ43151TEST' with 2 items: - item 1: type 'Books', collection 'Fiction' - item 2: type 'Music', collection 'Non-fiction' 3) Before rebuilding, confirm plack-error.log shows "Elasticsearch items index not ready" (if not, e.g. you already rebuilt earlier, skip to step 4). Then hit: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=ZZZ43151TEST&limit=itype:BK&limit=ccode:NFIC Should return a normal page, not an error - falling back to crash isn't. 4) Rebuild the items index: $ misc/search_tools/rebuild_elasticsearch.pl -i -r -v 5) Search '43' at http://localhost:8081/cgi-bin/koha/catalogue/search.pl 'Item types' facet shows both 'Books' and 'Music'. 6) Combine 'Books' + 'Fiction' (facets or &limit=itype:BK&limit, but the 'Item types' facet now shows only 'Books' (previously it kept showing 'Music' too). 7) Combine 'Books' + 'Non-fiction' (&limit=itype:BK&limit=ccode -> 0 results (previously incorrectly returned 1).8) Edit item 1 and item 2's home/holding library to two different libraries (allow ~15s for cache) - no rebuild needed. Combine an item type + one branch's facet: Verify our test biblio is returned 9) Run tests: prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202512|0 |1 is obsolete| | Attachment #202513|0 |1 is obsolete| | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 --- Comment #3 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 202690 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202690&action=edit Bug 43151: Require a single item to satisfy combined item-level search filters Combined item-level limits (item type, collection, location, home/holding library, availability) were checked independently against each bib's flattened, cross-item field arrays, so a bib could match a filter combination no single item actually satisfied (e.g. itype:BOOK + ccode:FICTION matching a bib whose only book is non-fiction and whose only fiction item is a CD). QueryBuilder now extracts these limits into constraint groups before they're folded into the query string, and Search resolves them as one combined query against the items index, requiring a single item to satisfy all of them together. Falls back to the existing (imperfect) query-string behaviour if the items index isn't ready. Note: this does not fix the item-level facet counts shown alongside these results, which are still built from the same flattened biblio-level arrays and can still list values that don't reflect the active combined filter. That's a separate, not yet addressed, follow-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Kristi Krueger <kkrueger@cuyahogalibrary.org> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 --- Comment #4 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 202691 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202691&action=edit Bug 43151: Scope item-level facet counts to the active combined filter Facet counts (itype, location, ccode, homebranch, holdingbranch) scanned every item on each matching bib, ignoring the active combined filter - filtering to itype:BOOK + ccode:FICTION still listed unrelated values like Music in the itype facet. _get_item_facets now runs one query per facet field, each excluding that field's own constraint but keeping the others, so facets stay accurate without hiding valid alternative values (standard multi-select faceting). Test plan: 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Catalogue a record titled (MARC21: 245$a) e.g. 'ZZZ43151TEST' with 2 items: - item 1: type 'Books', collection 'Fiction' - item 2: type 'Music', collection 'Non-fiction' 3) Before rebuilding, confirm plack-error.log shows "Elasticsearch items index not ready" (if not, e.g. you already rebuilt earlier, skip to step 4). Then hit: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=ZZZ43151TEST&limit=itype:BK&limit=ccode:NFIC Should return a normal page, not an error - falling back to crash isn't. 4) Rebuild the items index: $ misc/search_tools/rebuild_elasticsearch.pl -i -r -v 5) Search '43' at http://localhost:8081/cgi-bin/koha/catalogue/search.pl 'Item types' facet shows both 'Books' and 'Music'. 6) Combine 'Books' + 'Fiction' (facets or &limit=itype:BK&limit, but the 'Item types' facet now shows only 'Books' (previously it kept showing 'Music' too). 7) Combine 'Books' + 'Non-fiction' (&limit=itype:BK&limit=ccode -> 0 results (previously incorrectly returned 1).8) Edit item 1 and item 2's home/holding library to two different libraries (allow ~15s for cache) - no rebuild needed. Combine an item type + one branch's facet: Verify our test biblio is returned 9) Run tests: prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t Signed-off-by: Kristi Krueger <kkrueger@cuyahogalibrary.org> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Sponsorship status|--- |Sponsored Comma delimited| |Cuyahoga County Public list of Sponsors| |Library -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@openfifth.c | |o.uk --- Comment #5 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- There doesn't appear to be an end-to-end test through build_query_compat exercising this pipeline with realistic raw limits (e.g. mc-itype:BK, available, a branch-dropdown limit) and asserting $query->{_item_level_constraints} comes out right. Every test either calls _extract_item_level_constraints directly with synthetic strings, or calls _apply_item_level_filters/_get_matching_biblionumbers/_get_item_facets with hand-built group structures — the actual join between "what a real search sends" and "what gets extracted" is only verified manually per the commit's KTD test plan. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35052 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35052 [Bug 35052] OpacHiddenItemsHidesRecord system preference should be considered on index time instead -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |pedro.amorim@openfifth.co.u |ity.org |k Initiative type|--- |Feature -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |43017 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43017 [Bug 43017] [OMNIBUS] Interface optimization -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202690|0 |1 is obsolete| | Attachment #202691|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 --- Comment #6 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205601 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205601&action=edit Bug 43151: Require a single item to satisfy combined item-level search filters Combined item-level limits (item type, collection, location, home/holding library, availability) were checked independently against each bib's flattened, cross-item field arrays, so a bib could match a filter combination no single item actually satisfied (e.g. itype:BOOK + ccode:FICTION matching a bib whose only book is non-fiction and whose only fiction item is a CD). QueryBuilder now extracts these limits into constraint groups before they're folded into the query string, and Search resolves them as one combined query against the items index, requiring a single item to satisfy all of them together. Falls back to the existing (imperfect) query-string behaviour if the items index isn't ready. Note: this does not fix the item-level facet counts shown alongside these results, which are still built from the same flattened biblio-level arrays and can still list values that don't reflect the active combined filter. That's a separate, not yet addressed, follow-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Kristi Krueger <kkrueger@cuyahogalibrary.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 --- Comment #7 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205602 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205602&action=edit Bug 43151: Scope item-level facet counts to the active combined filter Facet counts (itype, location, ccode, homebranch, holdingbranch) scanned every item on each matching bib, ignoring the active combined filter - filtering to itype:BOOK + ccode:FICTION still listed unrelated values like Music in the itype facet. _get_item_facets now runs one query per facet field, each excluding that field's own constraint but keeping the others, so facets stay accurate without hiding valid alternative values (standard multi-select faceting). Test plan: 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Catalogue a record titled (MARC21: 245$a) e.g. 'ZZZ43151TEST' with 2 items: - item 1: type 'Books', collection 'Fiction' - item 2: type 'Music', collection 'Non-fiction' 3) Before rebuilding, confirm plack-error.log shows "Elasticsearch items index not ready" (if not, e.g. you already rebuilt earlier, skip to step 4). Then hit: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=ZZZ43151TEST&limit=itype:BK&limit=ccode:NFIC Should return a normal page, not an error - falling back to crash isn't. 4) Rebuild the items index: $ misc/search_tools/rebuild_elasticsearch.pl -i -r -v 5) Search '43' at http://localhost:8081/cgi-bin/koha/catalogue/search.pl 'Item types' facet shows both 'Books' and 'Music'. 6) Combine 'Books' + 'Fiction' (facets or &limit=itype:BK&limit, but the 'Item types' facet now shows only 'Books' (previously it kept showing 'Music' too). 7) Combine 'Books' + 'Non-fiction' (&limit=itype:BK&limit=ccode -> 0 results (previously incorrectly returned 1).8) Edit item 1 and item 2's home/holding library to two different libraries (allow ~15s for cache) - no rebuild needed. Combine an item type + one branch's facet: Verify our test biblio is returned 9) Run tests: prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t Signed-off-by: Kristi Krueger <kkrueger@cuyahogalibrary.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 --- Comment #8 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205603 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205603&action=edit Bug 43151: (QA follow-up) Stop a stale available:true token vetoing an accurate items-index filter _apply_item_level_filters left the biblio-level available:true token in the query string, reasoning that any bib passing the items-index filter would also satisfy that looser check. That's true for itype/ccode/ location/branch, but not for availability: Koha::Item::store (ITEM_CIRC_FIELDS) deliberately skips the biblio reindex on circulation-only changes once the items index is ready, so the biblios index's own available field is left stale on purpose. Since Elasticsearch ANDs must and filter clauses, that stale must-clause token could wrongly exclude a bib the fresh items-index filter had just matched - reopening the exact staleness bug Bug 42835's _apply_available_filter existed to fix. _strip_available_from_query_string restores that stripping (only for the available token, only once the items-index filter has been applied), so availability is resolved solely from the up-to-date items index again. Test plan: 1) prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 --- Comment #9 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205604 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205604&action=edit Bug 43151: (QA follow-up) Don't drop a branch constraint when two are present at once _extract_item_level_constraints stored the "neither homebranch nor holdingbranch" any_of branch shape in a single scalar, so a second branch-dropdown-shaped limit in the same search silently overwrote the first instead of being combined - quietly breaking the very "single item must satisfy every constraint" guarantee this feature exists for. Collect them into an array instead, so every any_of group extracted survives into the returned constraint list. Test plan: 1) prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 --- Comment #10 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 205605 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205605&action=edit Bug 43151: (QA follow-up) Add end-to-end build_query_compat test for item-level constraints Every existing test either called _extract_item_level_constraints directly with synthetic post-_fix_limit_special_cases strings, or called _apply_item_level_filters/_get_matching_biblionumbers/_get_item_facets with hand-built group structures - the join between what a real search sends and what gets extracted was only verified manually per the KTD test plan. Add a subtest that drives build_query_compat itself with raw limits as they'd actually arrive from a browser search (an mc-itype checkbox, the availability checkbox, and the "Individual libraries" branch dropdown under different SearchLimitLibrary settings, plus a phrase-quoted checkbox limit and non-item-level limits), asserting on the resulting $query->{_item_level_constraints}. Test plan: 1) prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |nick@bywatersolutions.com |y.org | -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org