[Bug 23649] New: Elasticsearch offer an option to weight search results by type
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Bug ID: 23649 Summary: Elasticsearch offer an option to weight search results by type Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com Some libraries would like to be able to prioritize "books" over "large print" , audios, etc in the search results -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Barbara Johnson <barbara.johnson@bedfordtx.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |barbara.johnson@bedfordtx.g | |ov --- Comment #1 from Barbara Johnson <barbara.johnson@bedfordtx.gov> --- Providing a way to organize the different formats for a title would provide an expected and reliable results display. It would improve searching for both patrons and staff. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Mwoo <mary.woodward@bedfordtx.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mary.woodward@bedfordtx.gov --- Comment #2 from Mwoo <mary.woodward@bedfordtx.gov> --- Assigning "weight" values to items would be a huge timesaver and alleviate a lot of patron frustrations. Our patrons will often put a hold on a large print book simply because it shows up first in their search. Invariably they didn't want the large print and are mad and disappointed. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #3 from Mwoo <mary.woodward@bedfordtx.gov> --- Assigning "weight" values to items would be a huge timesaver and alleviate a lot of patron frustrations. Our patrons will often put a hold on a large print book simply because it shows up first in their search. Invariably they didn't want the large print and are mad and disappointed. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Joy Nelson <joy@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joy@bywatersolutions.com Status|NEW |In Discussion -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Christopher Davis <cgdavis@uintah.utah.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cgdavis@uintah.utah.gov -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Björn Nylén <bjorn.nylen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bjorn.nylen@ub.lu.se --- Comment #4 from Björn Nylén <bjorn.nylen@ub.lu.se> --- Created attachment 146585 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=146585&action=edit [Do not push] - Experimental support for function_score We've been experimenting a bit with function_score query in our ES-searches as we're having the same problems with not very relevant records showing high in the results. The patch allows us to assign different function_score-functions to promote/demote records in the resultlist. It's a bit of work in progress but if someone wants to try it out: Apply patch Create a Local use syspref, ElasticsearchFunctionScore , textarea Fill in syspref with yaml representing av array of function scores as understood by ES. Eg(modify itype to local condidtions): - filter: match: itype: "3" weight: 1.5 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #5 from Björn Nylén <bjorn.nylen@ub.lu.se> --- More interesting things can be done eg. maths: In this case we want to promote newer books over older. So define a new numeric field 'pubyear-numeric' in the ES mappings and map it to 008/07-10. After some existance- and sanity-checking we weight records (in this case) newer than 1980 higher. - script_score: script: source: "long y = doc['pubyear-numeric'].size()==0 ? params.start : doc['pubyear-numeric'].value; y = (y<params.start || y>2100) ? params.start : y; return (y-params.start)/params.divider+1;" params: start: 1980 divider: 20 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Michael Adamyk <madamyk@ckls.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |madamyk@ckls.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #6 from Michael Adamyk <madamyk@ckls.org> --- This would be great! We often get random items like cake pans, kits, etc. near the top of our search results, which can be frustrating. More customization for search weights would be fantastic. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Steve, OSLRI, USA <sspohn@oslri.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bywater@oslri.net, | |sspohn@oslri.net -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 carthur@slolibrary.org <carthur@slolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |carthur@slolibrary.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #7 from carthur@slolibrary.org <carthur@slolibrary.org> --- This would be useful for SLO Libraries as well. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Chip Halvorsen <Chip.Halvorsen@WestlakeLibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |Chip.Halvorsen@WestlakeLibr | |ary.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Jason Robb <jrobb@sekls.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jrobb@sekls.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |domm@plix.at --- Comment #8 from Thomas Klausner <domm@plix.at> --- We would also be interested in this feature. I will aks the client if they want to fund it. If not, I will definitely QA it :-) -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Mathieu Saby <mathsabypro@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mathsabypro@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Lauren Denny <lauren_denny@sil.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lauren_denny@sil.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Sally <sally.healey@cheshiresharedservices.gov.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sally.healey@cheshireshared | |services.gov.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 George Williams (NEKLS) <george@nekls.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |george@nekls.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Michelle Spinney <mspinney@clamsnet.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mspinney@clamsnet.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |pedro.amorim@openfifth.co.u |ity.org |k CC| |pedro.amorim@openfifth.co.u | |k Status|In Discussion |ASSIGNED --- Comment #9 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Hi all, I'll be working on this. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #10 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200650 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200650&action=edit Bug 23649: Add search_field_value_boost DB table Adds a new table search_field_value_boost to store per-value relevance multipliers for facetable search fields (e.g. boost itype 'BK' by 2.0). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #11 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200651 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200651&action=edit Bug 23649: [DO NOT PUSH] DBIC schema for search_field_value_boost Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #12 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200652 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200652&action=edit Bug 23649: Add atomicupdate for search_field_value_boost table Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #13 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200653 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200653&action=edit Bug 23649: Add Koha::SearchFieldValueBoost ORM classes Adds Koha::SearchFieldValueBoost (singular) and Koha::SearchFieldValueBoosts (set) as thin Koha::Object wrappers around the search_field_value_boost table. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #14 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200654 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200654&action=edit Bug 23649: Apply function_score value boosts in ES QueryBuilder Wraps the biblios Elasticsearch query in a function_score query when any search_field_value_boost rows are configured. Each row becomes a filter+weight function: documents matching a boosted value have their relevance score multiplied by the configured weight; documents matching no filter are unaffected (neutral 1.0 multiplier). score_mode 'max' takes the highest matching weight per document; boost_mode 'multiply' applies it against the original query score. Boost functions are cached for 5 minutes under the elasticsearch_value_boost_functions key, cleared alongside the existing search fields cache when mappings are saved. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #15 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200655 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200655&action=edit Bug 23649: Add value boost configuration to ES mappings admin Adds a 'Relevance boosts' section to the biblios tab of the Elasticsearch mappings admin page. Librarians can add per-value weight multipliers for any facetable field (itype, location, ccode, etc.) and delete existing ones. Each entry is saved to search_field_value_boost and the boost cache is invalidated on save. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #16 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200656 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200656&action=edit Bug 23649: Add tests for function_score value boosts in QueryBuilder Tests _get_value_boost_functions and the function_score wrapping in build_query. Covers: empty arrayref when no boosts configured, correct filter/weight structure per boost, score_mode/boost_mode values, the original bool query preserved inside function_score, and authorities index not wrapped. Mocks _get_value_boost_functions at module level so existing build_query tests are unaffected by any value boosts present in the test DB or cache. 0) Launch ktd with elastic search: $ ktd --search-engine es8 up 1) Apply patches, updatedatabase and restart plack. 2) Do a search for 'music' and leave this tab available (so we can come back to it and compare relevance sorting): <staff_url>/cgi-bin/koha/catalogue/search.pl?q=music 3) Go to the Elasticsearch mappings admin page: <staff_url>/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl 4) Click the 'bibliographic records' tab. Confirm a 'Relevance boosts' section appears at the bottom with an empty table and an add form. 5) In the add form, select a facetable field (e.g. 'Item type'), enter value 'MU' and weight '2.00'. Click 'Add'. 6) Confirm the boost appears in the table with the correct field, value and weight. 7) Add a second boost on the same field (Item type) with value 'BK' and weight '0.5'. Click 'Add'. 8) Confirm both boosts are listed. 9) Perform the same 'music' search on a new tab: <staff_url>/cgi-bin/koha/catalogue/search.pl?q=music 10) Confirm that 'MU' (music) biblios now appear first. And BK (Books) only show more prominently towards the end of the results. 11) Click 'Delete' on the MU boost. Confirm it disappears from the table. 12) Perform the search again. Notice many 'MU' music biblios still show first, because BK (Books) are still being penalized (0.50 weight). But e.g. biblio 437 'Clothes, clothes, clothes' now appears in position #15 instead of position #20. 12) Go to the 'authorities' tab. Confirm no 'Relevance boosts' section appears there. 13) Try adding a new search field value boost with the '+Add' button with an empty field value. Confirm an error message appears and nothing is saved. 14) Try submitting with weight '0' or a negative number. Confirm a "must be a positive decimal" error appears. Note: boosts are applied at search time, not index time — changing or deleting a boost takes effect immediately with no reindexing required. prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Reset.t prove t/db_dependent/Koha/SearchField.t prove t/00-load.t Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Elasticsearch offer an |Elasticsearch offer an |option to weight search |option to weight search |results by type |results by value of facet | |field -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #17 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Hi all, I've widened the scope here a bit to allow configuring weight based on value of a facet field including, but not limited to, item type. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Jason Robb <jrobb@sekls.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #18 from Jason Robb <jrobb@sekls.org> --- Thanks for expanding this beyond item type! Overall the boosting seems to be working well. I had to skip over the first attachment (146585) to get things to work. The main issue I ran into is that the big yellow Save button below the table clears any boosts that you set. The Save button itself is misleadingly close to the boost settings so it is pretty easy to click it thinking it will save your boosts. UX-wise, adding weights is a little frustrating since it is at the bottom of a very long page and you can only do one at a time. Every time you add or delete a weight, you have to click the "Bibliographic record" tab again and scroll all the way down past the 370-row table. It would be great if the reload sent you back directly to the same area of the page. The placeholder text for the Value inputs "e.g. BK" is referencing an AV that may not exist in production systems since item type codes can vary. It would be handy if the Value column could use AV dropdowns for relevant Search fields to avoid input errors. I hit a few quirks while searching as well. Using a tilde operator with a test search of bird~ gave me 7 results. For both homebranch and holdingbranch I tried weighting MPL to 2 which put the Midway items at 1, 2 and 4 with a lone Fairview at 3. Weighting CPL puts Centerville items at 1, 2, 3 and 5 with the Fairview item at 4. The weighting does seem to be accurate in regular (no tilde) searches. When testing author weights, weighting a single author only worked if the punctuation was included (i.e. weighting "Aristophanes" doesn't work, but "Aristophanes." with the period does.) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Fiona Borthwick <fiona.borthwick@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fiona.borthwick@openfifth.c | |o.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Juliet Heltibridle <jheltibridle@rcplib.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jheltibridle@rcplib.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Jason Robb <jrobb@sekls.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Sponsorship status|--- |Sponsored Comma delimited| |Southeast Kansas Library list of Sponsors| |System -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Jason Robb <jrobb@sekls.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #24 from Jason Robb <jrobb@sekls.org> --- Thanks Pedro! Applying the patch gets hung up on the first attachment still (146585 - [Do not push] - Experimental support for function_score) but if I skip it everything seems to be working -- does that need obsoleted? I'm also still seeing an issue with boosts getting completely removed when other changes are made on the page. To replicate, add some boosts, navigate away and back to the page, boosts are still there. Make changes to something else (i.e. rearrange facet order or add a search field weight on the first tab) then click the yellow "Save" button. Boosts get wiped out. I'm still fiddling around with testing boosts and search weights but I wanted to make sure those things didn't get lost. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #25 from Jason Robb <jrobb@sekls.org> --- In testing the searching with the JS from bug 42107 -- $(".search_score").toggle(); -- and adding some weights via the search field settings, things don't seem to be multiplying as I would expect in a way that would prioritize one facet over another. For example, if I give: - title a Search field weight of 32 - itype a Search field weight of 20 - itype = MU a Relevance boost of 2 - itype = BK a Relevance boost of 0.5 A search for "sugar" will return 3 results. The first two get a score of 64 for a (title match 32 * boost 2). The third result does not have a title match but does have sugar in the record so it only gets a score of 2 (from boost 2). So the title weight is getting multiplied rather than the item type weight in those scenarios. If I add a book with "sugar" in the title to the mix, it gets a score of 16 (title weight 32 * boost 0.5) so it lands above the third music item. Could (facet weight * facet boost) be added to the existing search field weight instead? That would lead to results in order in this example of - Sugar music 1: 32 +(20*2) = 72 - Sugar music 2: 32 +(20*2) = 72 - Sugar-adjacent music: (20*2) = 40 - Sugar book: (32*0.5) = 16 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 koha-US bug tracker <bugzilla@koha-us.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bugzilla@koha-us.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Elizabeth Hoffman <ehoffman@plumcreeklibrary.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ehoffman@plumcreeklibrary.n | |et -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact| |testopia@bugs.koha-communit | |y.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #146585|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #26 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 205059 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=205059&action=edit Bug 23649: Preserve value boosts when saving search field mappings Saving the mappings form deletes and recreates every search_field row, cascading away all search_field_value_boost rows via its ON DELETE CASCADE. Koha::SearchFieldValueBoosts gains existing_boosts_by_search_field_name/restore_from_existing_boosts to preserve boosts across the rebuild. Also adds the missing Koha::SearchFieldValueBoost->search_field accessor. prove t/db_dependent/Koha/SearchFieldValueBoost.t Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #27 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- (In reply to Jason Robb from comment #24)
Thanks Pedro!
Applying the patch gets hung up on the first attachment still (146585 - [Do not push] - Experimental support for function_score) but if I skip it everything seems to be working -- does that need obsoleted?
I'm also still seeing an issue with boosts getting completely removed when other changes are made on the page.
To replicate, add some boosts, navigate away and back to the page, boosts are still there. Make changes to something else (i.e. rearrange facet order or add a search field weight on the first tab) then click the yellow "Save" button. Boosts get wiped out.
I'm still fiddling around with testing boosts and search weights but I wanted to make sure those things didn't get lost.
Hi Jason, thank you for this detailed feedback. Addressed in the last follow-up. (In reply to Jason Robb from comment #25)
In testing the searching with the JS from bug 42107 -- $(".search_score").toggle(); -- and adding some weights via the search field settings, things don't seem to be multiplying as I would expect in a way that would prioritize one facet over another.
For example, if I give: - title a Search field weight of 32 - itype a Search field weight of 20 - itype = MU a Relevance boost of 2 - itype = BK a Relevance boost of 0.5
A search for "sugar" will return 3 results. The first two get a score of 64 for a (title match 32 * boost 2). The third result does not have a title match but does have sugar in the record so it only gets a score of 2 (from boost 2). So the title weight is getting multiplied rather than the item type weight in those scenarios.
If I add a book with "sugar" in the title to the mix, it gets a score of 16 (title weight 32 * boost 0.5) so it lands above the third music item.
Could (facet weight * facet boost) be added to the existing search field weight instead? That would lead to results in order in this example of - Sugar music 1: 32 +(20*2) = 72 - Sugar music 2: 32 +(20*2) = 72 - Sugar-adjacent music: (20*2) = 40 - Sugar book: (32*0.5) = 16
I understand we sometimes want a category to outrank a weak text match, i.e. "always show music prominently for this search" is a reasonable thing to want. But we can already do that under multiply, no field weight needed. The boost's own weight is ours to set, we can just set it higher. You used 2 in your test, a mild nudge, exactly what the hint text promises. If we set MU's boost to 20 instead, the "Unconditional" bib record (i.e. 'sugar-adjacent record) scores 1 × 20 = 20, still proportional, still consistent across queries, and now beats the book's 16. Same outcome you wanted, existing knob, no side effects. It's not really multiply vs sum, it's whether the boost weight gives you enough range. It does, up to 999.99. Nothing stops us from setting a boost of 50 if we want something to basically always win. Where sum falls short: +40 only beat 16 because this search happened to produce small base scores (32, ~1). Run a search where scores naturally land in the hundreds, and that same +40 does nothing, buried under real relevance. Run one where scores are tiny, and +40 flattens every result to basically the same rank regardless of actual match quality. We can't predict which you'll get, so a boost of 2 doesn't have a consistent meaning. multiply doesn't have this problem, doubling always means doubling, no matter what search you run. I believe we should keep multiply, we can already get what we're after by raising the boost weight itself. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Jason Robb <jrobb@sekls.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #28 from Jason Robb <jrobb@sekls.org> --- Weight values over 999 fail and throw this console error: mappings.pl:44363 POST http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings... 500 (Internal Server Error) (anonymous) @ mappings.pl:44363 dispatch @ jquery-3.6.0.min_26.0600026.js:2 v.handle @ jquery-3.6.0.min_26.0600026.js:2 VM937:1 Uncaught (in promise) SyntaxError: Unexpected token 'D', "DBI Except"... is not valid JSON Thanks for the clarification on the math! If I'm only boosting a single search field type (i.e. all my boosts are on itype), things seem to be working fairly well! I did run into a possible issue when boosting multiple different search field types. It appears to only take one into account. As an example: 1.) Add $(".search_score").toggle(); to IntranetUserJS to see the search score 2.) Search for john > results all score 1 3.) Add a boost of itype > BK > 200 4.) Search for john > records with BK in the 942$c get a score of 200 (some of the results look wrong because of the item-level itypes but the record-level types are boosted correctly) 5.) Add an additional boost of location > GEN > 200 6.) Search for john. The fourth result has BK in the 942$c but the items have locations of AV, and there are additional results after it where both the itype is BK and the locations are GEN. Shouldn't it be putting results where multiple boosts match higher than results that only match some of the boosts? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #29 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 206584 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206584&action=edit Bug 23649: (QA follow-up) Reject relevance boost weights over 999.99 Weight was only validated for > 0, not the decimal(5,2) column's 999.99 cap. Values over it threw an uncaught DBI exception on store, returning a raw, unhandled 500. Test plan: 1. Add a boost with weight '1000'. Confirm a validation error, no 500. 2. Add a boost with weight '999.99'. Confirm it saves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #30 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 206585 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=206585&action=edit Bug 23649: (QA follow-up) Sum overlapping relevance boosts instead of taking the max score_mode was 'max': a record matching multiple boosts only counted the single highest weight. Now 'sum', so matching weights add up. Test plan (Jason's repro from comment 28, retested against the fix): 1.) Add $(".search_score").toggle(); to IntranetUserJS to see the search score 2.) Search for john > results all score 1 3.) Add a boost of itype > BK > 200 4.) Search for john > records with BK in the 942$c get a score of 200 (some of the results look wrong because of the item-level itypes but the record-level types are boosted correctly) 5.) Add an additional boost of location > GEN > 200 6.) Search for john. Confirm records with both itype BK and location GEN now score 400, ranking above records matching only one of the two boosts (score 200). prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23649 --- Comment #31 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- (In reply to Jason Robb from comment #28)
Weight values over 999 fail and throw this console error:
mappings.pl:44363 POST http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings. pl 500 (Internal Server Error) (anonymous) @ mappings.pl:44363 dispatch @ jquery-3.6.0.min_26.0600026.js:2 v.handle @ jquery-3.6.0.min_26.0600026.js:2 VM937:1 Uncaught (in promise) SyntaxError: Unexpected token 'D', "DBI Except"... is not valid JSON
Thanks for the clarification on the math!
If I'm only boosting a single search field type (i.e. all my boosts are on itype), things seem to be working fairly well!
I did run into a possible issue when boosting multiple different search field types. It appears to only take one into account. As an example:
1.) Add $(".search_score").toggle(); to IntranetUserJS to see the search score 2.) Search for john > results all score 1 3.) Add a boost of itype > BK > 200 4.) Search for john > records with BK in the 942$c get a score of 200 (some of the results look wrong because of the item-level itypes but the record-level types are boosted correctly) 5.) Add an additional boost of location > GEN > 200 6.) Search for john. The fourth result has BK in the 942$c but the items have locations of AV, and there are additional results after it where both the itype is BK and the locations are GEN.
Shouldn't it be putting results where multiple boosts match higher than results that only match some of the boosts?
Thanks for testing Jason, I believe the last 2 patches address these. Please re test when convenient. Thanks! -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org