[Bug 39079] New: Matchpoints with multiple fields require all fields to match under Elasticsearch
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Bug ID: 39079 Summary: Matchpoints with multiple fields require all fields to match under Elasticsearch Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org In zebra when matching an incoming file to existing records, we use 'OR' to combine the fields In elasticsearch we are combining multiple fields with 'AND' as a 'must' search This means an incoming record with two control numbers will not match an existing record with only one The generated queries are like: 2025-02-10T14:59:07 zebrasrv(46) [request] Search authorities OK 0 1 1+0 RPN @attrset Bib-1 @or @attr 1=Local-Number @attr 4=109 @attr 7=1 0 @or @or @attr 6=3 @attr 5=100 @attr 4=1 @attr 1=other-control-number (DE-588)4179998-7 @attr 6=3 @attr 5=100 @attr 4=1 @attr 1=other-control-number (DE-627)104628669 @attr 6=3 @attr 5=100 @attr 4=1 @attr 1=other-control-number (DE-576)209995106 versus 'bool' => { 'must' => [ { 'term' => { 'other-control-number.ci_raw' => '(DE-588)4179998-7' } }, { 'term' => { 'other-control-number.ci_raw' => '(DE-627)104628669' } }, { 'term' => { 'other-control-number.ci_raw' => '(DE-576)209995106' } } ] } } This is due to this bit of code: ElasticSearch/QueryBuilder: 469 # Merge the query parts appropriately 470 # 'should' behaves like 'or' 471 # 'must' behaves like 'and' 472 # Zebra behaviour seem to match must so using that here 473 my $elastic_query = {}; 474 $elastic_query->{bool}->{must} = \@query_parts; We need to respect the and_or passed into build_auth_query_compat, but currently: 514 =item and_or 515 516 Totally ignored. It is never used in L<C4::AuthoritiesMarc::SearchAuthorities>. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 --- Comment #1 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 177706 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=177706&action=edit Bug 39079: Follow 'and_or' variable when passed to build_auth_query_compat When building queries for matchign Zebra uses OR to combine searches - Elasticsearch has been effectively using 'AND' - this patch checks whether and_or has been passed and follows the variable when present. To test: 1 - Have an import file of auths with multiple 035 2 - Setup an authority matching rule Search index: other-control-number Score: 1000 Tag: 035 Subfields: a 3 - Make sure Koha is using Elasticsearch 4 - Stage file without matching, import file 5 - Stage file again, using your matching rule 6 - Note all records match 7 - Setup Cataloging->Marc modification template 8 - Add an action to delete first 035 field 9 - Run it against the records you just imported You can use a report: SELECT authid FROM auth_header WHERE authid > #### (find the lowest new record number in batch you imported) 10 - Change matching rule in batch to none, then back to your rule 11 - No matches! 12 - Apply patch, restart all 13 - Set matchign rule to none, then back to your rule 14 - Matches! -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |nick@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pascal.uphaus@gwdg.de -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply CC| |magnus@libriotech.no --- Comment #2 from Magnus Enger <magnus@libriotech.no> --- $ git bz apply 39079 Bug 39079 - Matchpoints with multiple fields require all fields to match under Elasticsearch 177706 - Bug 39079: Follow 'and_or' variable when passed to build_auth_query_compat Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 39079: Follow 'and_or' variable when passed to build_auth_query_compat error: sha1 information is lacking or useless (Koha/SearchEngine/Elasticsearch/QueryBuilder.pm). error: could not build fake ancestor Patch failed at 0001 Bug 39079: Follow 'and_or' variable when passed to build_auth_query_compat hint: Use 'git am --show-current-patch=diff' to see the failed patch When you have resolved this problem run "git bz apply --continue". If you would prefer to skip this patch, instead run "git bz apply --skip". To restore the original branch and stop patching run "git bz apply --abort". Patch left in /tmp/Bug-39079-Follow-andor-variable-when-passed-to-bui-W7J4WR.patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #177706|0 |1 is obsolete| | --- Comment #3 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 180108 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=180108&action=edit Bug 39079: Follow 'and_or' variable when passed to build_auth_query_compat When building queries for matchign Zebra uses OR to combine searches - Elasticsearch has been effectively using 'AND' - this patch checks whether and_or has been passed and follows the variable when present. To test: 1 - Have an import file of auths with multiple 035 2 - Setup an authority matching rule Search index: other-control-number Score: 1000 Tag: 035 Subfields: a 3 - Make sure Koha is using Elasticsearch 4 - Stage file without matching, import file 5 - Stage file again, using your matching rule 6 - Note all records match 7 - Setup Cataloging->Marc modification template 8 - Add an action to delete first 035 field 9 - Run it against the records you just imported You can use a report: SELECT authid FROM auth_header WHERE authid > #### (find the lowest new record number in batch you imported) 10 - Change matching rule in batch to none, then back to your rule 11 - No matches! 12 - Apply patch, restart all 13 - Set matchign rule to none, then back to your rule 14 - Matches! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #180108|0 |1 is obsolete| | --- Comment #4 from Magnus Enger <magnus@libriotech.no> --- Created attachment 180151 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=180151&action=edit Bug 39079: Follow 'and_or' variable when passed to build_auth_query_compat When building queries for matchign Zebra uses OR to combine searches - Elasticsearch has been effectively using 'AND' - this patch checks whether and_or has been passed and follows the variable when present. To test: 1 - Have an import file of auths with multiple 035 2 - Setup an authority matching rule Search index: other-control-number Score: 1000 Tag: 035 Subfields: a 3 - Make sure Koha is using Elasticsearch 4 - Stage file without matching, import file 5 - Stage file again, using your matching rule 6 - Note all records match 7 - Setup Cataloging->Marc modification template 8 - Add an action to delete first 035 field 9 - Run it against the records you just imported You can use a report: SELECT authid FROM auth_header WHERE authid > #### (find the lowest new record number in batch you imported) 10 - Change matching rule in batch to none, then back to your rule 11 - No matches! 12 - Apply patch, restart all 13 - Set matchign rule to none, then back to your rule 14 - Matches! Signed-off-by: Magnus Enger <magnus@libriotech.no> Works as advertised! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |martin.renvoize@openfifth.c |y.org |o.uk CC| |martin.renvoize@openfifth.c | |o.uk -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #180151|0 |1 is obsolete| | --- Comment #5 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 182263 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=182263&action=edit Bug 39079: Follow 'and_or' variable when passed to build_auth_query_compat When building queries for matchign Zebra uses OR to combine searches - Elasticsearch has been effectively using 'AND' - this patch checks whether and_or has been passed and follows the variable when present. To test: 1 - Have an import file of auths with multiple 035 2 - Setup an authority matching rule Search index: other-control-number Score: 1000 Tag: 035 Subfields: a 3 - Make sure Koha is using Elasticsearch 4 - Stage file without matching, import file 5 - Stage file again, using your matching rule 6 - Note all records match 7 - Setup Cataloging->Marc modification template 8 - Add an action to delete first 035 field 9 - Run it against the records you just imported You can use a report: SELECT authid FROM auth_header WHERE authid > #### (find the lowest new record number in batch you imported) 10 - Change matching rule in batch to none, then back to your rule 11 - No matches! 12 - Apply patch, restart all 13 - Set matchign rule to none, then back to your rule 14 - Matches! Signed-off-by: Magnus Enger <magnus@libriotech.no> Works as advertised! Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #6 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Makes sense, follows Zebra precedent. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |25.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 --- Comment #7 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 25.05! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Status|Pushed to main |Pushed to stable Version(s)|25.05.00 |25.05.00,24.11.10 released in| | --- Comment #8 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 24.11.x for 24.11.10 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39079 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Needs documenting --- Comment #9 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- Not backporting to 22.11.x as it wasn't backported to 24.05.x -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org