[Bug 43151] New: ES should apply search filters on each item discretely
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Bug ID: 43151 Summary: ES should apply search filters on each item discretely Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: andrew@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org Depends on: 42835 Target Milestone: 26.11 When filtering search results using item record values, Koha checks for each of those values at the biblio level, rather than at the item level. This leads to undesirable search results. For example, assume you have a biblio with two items: - item 1 is itype BOOK and collection FICTION - item 2 is itype AUDIO and collection NONFICTION Currently, if a user performs a search limited to itype=BOOK and collection=NONFICTION, this biblio will be returned because it has at least one BOOK item and at least one NONFICTION item. But the user expectation is that the search will return biblios where at least one item that is both a BOOK and NONFICTION. Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42835 [Bug 42835] ElasticSearch should have an items index -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 --- Comment #1 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202512 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202512&action=edit Bug 43151: Require a single item to satisfy combined item-level search filters Combined item-level limits (item type, collection, location, home/holding library, availability) were checked independently against each bib's flattened, cross-item field arrays, so a bib could match a filter combination no single item actually satisfied (e.g. itype:BOOK + ccode:FICTION matching a bib whose only book is non-fiction and whose only fiction item is a CD). QueryBuilder now extracts these limits into constraint groups before they're folded into the query string, and Search resolves them as one combined query against the items index, requiring a single item to satisfy all of them together. Falls back to the existing (imperfect) query-string behaviour if the items index isn't ready. Note: this does not fix the item-level facet counts shown alongside these results, which are still built from the same flattened biblio-level arrays and can still list values that don't reflect the active combined filter. That's a separate, not yet addressed, follow-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 --- Comment #2 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 202513 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202513&action=edit Bug 43151: Scope item-level facet counts to the active combined filter Facet counts (itype, location, ccode, homebranch, holdingbranch) scanned every item on each matching bib, ignoring the active combined filter - filtering to itype:BOOK + ccode:FICTION still listed unrelated values like Music in the itype facet. _get_item_facets now runs one query per facet field, each excluding that field's own constraint but keeping the others, so facets stay accurate without hiding valid alternative values (standard multi-select faceting). Test plan: 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Catalogue a record titled (MARC21: 245$a) e.g. 'ZZZ43151TEST' with 2 items: - item 1: type 'Books', collection 'Fiction' - item 2: type 'Music', collection 'Non-fiction' 3) Before rebuilding, confirm plack-error.log shows "Elasticsearch items index not ready" (if not, e.g. you already rebuilt earlier, skip to step 4). Then hit: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=ZZZ43151TEST&limit=itype:BK&limit=ccode:NFIC Should return a normal page, not an error - falling back to crash isn't. 4) Rebuild the items index: $ misc/search_tools/rebuild_elasticsearch.pl -i -r -v 5) Search '43' at http://localhost:8081/cgi-bin/koha/catalogue/search.pl 'Item types' facet shows both 'Books' and 'Music'. 6) Combine 'Books' + 'Fiction' (facets or &limit=itype:BK&limit, but the 'Item types' facet now shows only 'Books' (previously it kept showing 'Music' too). 7) Combine 'Books' + 'Non-fiction' (&limit=itype:BK&limit=ccode -> 0 results (previously incorrectly returned 1).8) Edit item 1 and item 2's home/holding library to two different libraries (allow ~15s for cache) - no rebuild needed. Combine an item type + one branch's facet: Verify our test biblio is returned 9) Run tests: prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #202512|0 |1 is obsolete| | Attachment #202513|0 |1 is obsolete| | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 --- Comment #3 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 202690 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202690&action=edit Bug 43151: Require a single item to satisfy combined item-level search filters Combined item-level limits (item type, collection, location, home/holding library, availability) were checked independently against each bib's flattened, cross-item field arrays, so a bib could match a filter combination no single item actually satisfied (e.g. itype:BOOK + ccode:FICTION matching a bib whose only book is non-fiction and whose only fiction item is a CD). QueryBuilder now extracts these limits into constraint groups before they're folded into the query string, and Search resolves them as one combined query against the items index, requiring a single item to satisfy all of them together. Falls back to the existing (imperfect) query-string behaviour if the items index isn't ready. Note: this does not fix the item-level facet counts shown alongside these results, which are still built from the same flattened biblio-level arrays and can still list values that don't reflect the active combined filter. That's a separate, not yet addressed, follow-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Kristi Krueger <kkrueger@cuyahogalibrary.org> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 --- Comment #4 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 202691 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=202691&action=edit Bug 43151: Scope item-level facet counts to the active combined filter Facet counts (itype, location, ccode, homebranch, holdingbranch) scanned every item on each matching bib, ignoring the active combined filter - filtering to itype:BOOK + ccode:FICTION still listed unrelated values like Music in the itype facet. _get_item_facets now runs one query per facet field, each excluding that field's own constraint but keeping the others, so facets stay accurate without hiding valid alternative values (standard multi-select faceting). Test plan: 1) Set up KTD with Elasticsearch 8: ktd --search-engine es8 up 2) Catalogue a record titled (MARC21: 245$a) e.g. 'ZZZ43151TEST' with 2 items: - item 1: type 'Books', collection 'Fiction' - item 2: type 'Music', collection 'Non-fiction' 3) Before rebuilding, confirm plack-error.log shows "Elasticsearch items index not ready" (if not, e.g. you already rebuilt earlier, skip to step 4). Then hit: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=ZZZ43151TEST&limit=itype:BK&limit=ccode:NFIC Should return a normal page, not an error - falling back to crash isn't. 4) Rebuild the items index: $ misc/search_tools/rebuild_elasticsearch.pl -i -r -v 5) Search '43' at http://localhost:8081/cgi-bin/koha/catalogue/search.pl 'Item types' facet shows both 'Books' and 'Music'. 6) Combine 'Books' + 'Fiction' (facets or &limit=itype:BK&limit, but the 'Item types' facet now shows only 'Books' (previously it kept showing 'Music' too). 7) Combine 'Books' + 'Non-fiction' (&limit=itype:BK&limit=ccode -> 0 results (previously incorrectly returned 1).8) Edit item 1 and item 2's home/holding library to two different libraries (allow ~15s for cache) - no rebuild needed. Combine an item type + one branch's facet: Verify our test biblio is returned 9) Run tests: prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t Signed-off-by: Kristi Krueger <kkrueger@cuyahogalibrary.org> -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Sponsorship status|--- |Sponsored Comma delimited| |Cuyahoga County Public list of Sponsors| |Library -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43151 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@openfifth.c | |o.uk --- Comment #5 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- There doesn't appear to be an end-to-end test through build_query_compat exercising this pipeline with realistic raw limits (e.g. mc-itype:BK, available, a branch-dropdown limit) and asserting $query->{_item_level_constraints} comes out right. Every test either calls _extract_item_level_constraints directly with synthetic strings, or calls _apply_item_level_filters/_get_matching_biblionumbers/_get_item_facets with hand-built group structures — the actual join between "what a real search sends" and "what gets extracted" is only verified manually per the commit's KTD test plan. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org