[Bug 27597] New: Searching "kw:term" does not work with Elasticsearch
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Bug ID: 27597 Summary: Searching "kw:term" does not work with Elasticsearch Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: fridolin.somers@biblibre.com In Koha/SearchEngine/Elasticsearch/QueryBuilder.pm : There is a map of convertion from CCL : our %index_field_convert = ( 'kw' => '', 'ab' => 'abstract', 'au' => 'author', ... You see that kw is replaced by empty. See that in _convert_index_strings() : my ($conv) = $self->_convert_index_fields($field); unless ( defined($conv) ) { push @res, $s; next; } push @res, ($conv->{field} ? $conv->{field} . ':' : '') . $self->_modify_string_by_type( %$conv, operand => $term ); If Field is empty the character ":" is not added. This behavior is missing from _convert_index_strings_freeform() : $search =~ s/($field_name_pattern)($multi_field_pattern):/(exists $index_field_convert{$1} ? $index_field_convert{$1} : $1)."$2:"/oge; Query "kw:term" will be converted to ":term" :( -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=26537 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |didier.gautheron@biblibre.c | |om -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |alex.arnaud@biblibre.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 --- Comment #1 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 116241 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=116241&action=edit Bug 27597: Remove leading colon in ES query If we are searching on kw there is a leading colon at the beginning of the generated query: kw:foo becomes :foo Note that it only happens when there is no other terms before. Test plan: 0. Don't apply the patch 1. Search for kw:foo 2. Notice the error Error: Unable to perform your search. Please try again. and the logs say Failed to parse query [(:foo*)] 3. Apply the patch 4. Repeat the search and notice that you know get: "12 result(s) found for 'kw:foo'." -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 --- Comment #2 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 116242 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=116242&action=edit Bug 27597: Remove extra spaces in the generated query Look at the tests, sometime we have extra spaces. With this patch they still pass and the expected queries are better. Feel free to drop this patch if you have any concerns about it. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |jonathan.druart@bugs.koha-c |ity.org |ommunity.org CC| |jonathan.druart@bugs.koha-c | |ommunity.org -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 --- Comment #3 from Fridolin Somers <fridolin.somers@biblibre.com> --- Super, But we also need to fix in _convert_index_strings_freeform() rigth ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 --- Comment #4 from Fridolin Somers <fridolin.somers@biblibre.com> --- And BTW is it in theory correct to replace "kw:" with "*:" ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 --- Comment #5 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Fridolin Somers from comment #3)
Super,
But we also need to fix in _convert_index_strings_freeform() rigth ?
I fixed it where we deal with colons, so it seems to be the right place. _convert_index_strings_freeform is only called from _clean_search_term anyway. (In reply to Fridolin Somers from comment #4)
And BTW is it in theory correct to replace "kw:" with "*:" ?
If it's what we want we should map is in %index_field_convert then. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |major --- Comment #6 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- It also breaks itemBarcodeFallbackSearch. [2021/02/17 08:36:15] [WARN] [Request] ** [http://es:9200]-[400] [query_shard_exception] Failed to parse query [(:street*)], with: {"index_uuid":"YeYOreb1Rl2j-l52RVXJ4A","index":"koha_kohadev_biblios"}, called f rom sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Search.pm line 96. With vars: {'status_code' => 400,'body' => {'status' => 400,'error' => {'faile d_shards' => [{'index' => 'koha_kohadev_biblios','reason' => {'type' => 'query_shard_exception','index_uuid' => 'YeYOreb1Rl2j-l52RVXJ4A','caused_by' => {'reason' => 'Cannot parse \'(:street*)\': Encountered " ":" ": "" at line 1, column 1. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 --- Comment #7 from Fridolin Somers <fridolin.somers@biblibre.com> --- # Remove unquoted colons that have whitespace on either side of them - $term =~ s/(:+)(\s+)$lookahead/$2/g; + $term =~ s/(:+)(\s+)$lookahead//g; $term =~ s/(\s+)(:+)$lookahead/$1/g; Could you explain this change please ? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 --- Comment #8 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Fridolin Somers from comment #7)
# Remove unquoted colons that have whitespace on either side of them - $term =~ s/(:+)(\s+)$lookahead/$2/g; + $term =~ s/(:+)(\s+)$lookahead//g; $term =~ s/(\s+)(:+)$lookahead/$1/g;
Could you explain this change please ?
Look at the unit tests, it removes extra spaces. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Trivial patch Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #116241|0 |1 is obsolete| | --- Comment #9 from Fridolin Somers <fridolin.somers@biblibre.com> --- Created attachment 117856 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=117856&action=edit Bug 27597: Remove leading colon in ES query If we are searching on kw there is a leading colon at the beginning of the generated query: kw:foo becomes :foo Note that it only happens when there is no other terms before. Test plan: 0. Don't apply the patch 1. Search for kw:foo 2. Notice the error Error: Unable to perform your search. Please try again. and the logs say Failed to parse query [(:foo*)] 3. Apply the patch 4. Repeat the search and notice that you know get: "12 result(s) found for 'kw:foo'." Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #116242|0 |1 is obsolete| | --- Comment #10 from Fridolin Somers <fridolin.somers@biblibre.com> --- Created attachment 117857 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=117857&action=edit Bug 27597: Remove extra spaces in the generated query Look at the tests, sometime we have extra spaces. With this patch they still pass and the expected queries are better. Feel free to drop this patch if you have any concerns about it. Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 --- Comment #11 from Fridolin Somers <fridolin.somers@biblibre.com> --- All is good ;) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 --- Comment #12 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I am stuck on this one as I can't get Elasticsearch to work :( Error: Unable to perform your search. Please try again. Applied the patch, reset mappings, reindexed... status of Elasticsearch is green. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=24372 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Marjorie Barry-Vila <marjorie.barry-vila@collecto.ca> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |marjorie.barry-vila@collect | |o.ca -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #117856|0 |1 is obsolete| | Attachment #117857|0 |1 is obsolete| | --- Comment #13 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 118190 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=118190&action=edit Bug 27597: Remove leading colon in ES query If we are searching on kw there is a leading colon at the beginning of the generated query: kw:foo becomes :foo Note that it only happens when there is no other terms before. Test plan: 0. Don't apply the patch 1. Search for kw:foo 2. Notice the error Error: Unable to perform your search. Please try again. and the logs say Failed to parse query [(:foo*)] 3. Apply the patch 4. Repeat the search and notice that you know get: "12 result(s) found for 'kw:foo'." Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 --- Comment #14 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Fridolin Somers from comment #10)
Created attachment 117857 [details] [review] Bug 27597: Remove extra spaces in the generated query
Look at the tests, sometime we have extra spaces. With this patch they still pass and the expected queries are better. Feel free to drop this patch if you have any concerns about it.
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
I obsoleted this, it causes a recreation of issues solved in bug 24567, tested with the comment from bug 24372: Using an example record in the testing DB, try searching for: Pictura murală din nordul Moldovei: modificari estetice si restarare = Mural painting in the north of Moldavia: aesthetic modification and restoration In ES the search fails. If you look at the query we end up with terms like: Moldoveimodificari AND Moldaviaaesthetic -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |21.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 --- Comment #15 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 21.05, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|21.05.00 |21.05.00,20.11.04 released in| | Status|Pushed to master |Pushed to stable --- Comment #16 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 20.11.x for 20.11.04 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|21.05.00,20.11.04 |21.05.00,20.11.04,20.05.10 released in| | Status|Pushed to stable |Pushed to oldstable CC| |andrew@bywatersolutions.com --- Comment #17 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Pushed to 20.05.x for 20.05.10 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27597 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to oldstable |RESOLVED CC| |victor@tuxayo.net Resolution|--- |FIXED --- Comment #18 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Can't backport to 19.11.x: can't solve a conflict. If there is an interest in having this backported, please submit a patch for 19.11. <<<<<<< HEAD $term =~ s/(\:[:\s]+|[:\s]+:)$lookahead//g; ||||||| parent of 622a68f695 (Bug 27597: Remove leading colon in ES query) $term =~ s/(:+)(\s+)$lookahead/$2/g; $term =~ s/(\s+)(:+)$lookahead/$1/g; ======= $term =~ s/(:+)(\s+)$lookahead/$2/g; $term =~ s/(\s+)(:+)$lookahead/$1/g; $term =~ s/^://;
> 622a68f695 (Bug 27597: Remove leading colon in ES query)
-- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org