https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 Noémie Labine <noemie.labine@collecto.ca> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |noemie.labine@collecto.ca -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 Liz Rea <wizzyrea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wizzyrea@gmail.com --- Comment #2 from Liz Rea <wizzyrea@gmail.com> --- They've already fixed the exclamation point on bug 28316 :) -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #3 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I found this explanation about the question mark: The question mark is a special character in es query strings. For example, if you search "me?t", this means that the results we seek must contain a word that contain "At Most" one character between "me" and "t". So the words "meet", "meat" or "met" will match the query. So maybe a fix is not so easy, as we either make it ignore ? or keep the feature? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 Emmanuel Bétemps <e.betemps@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |e.betemps@gmail.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff Assignee|koha-bugs@lists.koha-commun |januszop@gmail.com |ity.org | --- Comment #6 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 147161 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=147161&action=edit [PATCH] Bug 28884: ElasticSearch: Question mark in title search returns no results 1. It happens that question mark (‘?’) is contained in bibliographic data (e.g. in title). 2. It happens that user perform searches with that contain question mark (e.g. searching with a title taken verbatim). 3. In ES queries, in particular in query_string query, question mark has a special meaning – substitutes one character (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-qu...). Question mark has also special meaning in regexp inside quiery_string (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-qu...). 4. Ordinary users are not aware of this special function of the question mark and, as a result, receive no results (e.g. while searching for title like "Will libraries survive? : the future of public libraries"). Test plan: ========== 1. Have biblio record with title like 245 10 $a Will libraries survive? : $b the future of public libraries. (= with question mark attached to a word) in Koha instance with ES. 2. Search for: Will libraries survive? the future of public libraries You should get no results. 3. Apply the patch, controll that QueryRegexEscapeOptions is set to Escape. 4. Repeat the search. You should get the record. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #8 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Nick Clemens from comment #7)

This was "solved" with a hack on bug 31213

If I understand what this patch is doing, you are escaping the question mark so it no longer operates as a special character?

This is not a bad idea, however, it needs to be documented with a comment in the code, as well as in the system preference. I think it would actually be better as its own system preference.

Nick, I'm aware of your 'hack'--now, I'm unable to verify if quickly, but as far as I remember it didn't solve this case. If it did, there would be no point for me to dig into this. If I remember well, the patch intends to treat the question mark, in the function _query_regex_escape_process, in exactly the same way as it is done with slash: instead of escaping/unescaping only '/': (?=/) it escapes also '?': (?=[/\?]). Maybe I could extend the existing comment, but I don't feel like this is a candidate for a new syspref...? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #10 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Katrin Fischer from comment #9)

(In reply to Janusz Kaczmarek from comment #8)

...

(In reply to Nick Clemens from comment #7)

...

This was "solved" with a hack on bug 31213

If I understand what this patch is doing, you are escaping the question mark so it no longer operates as a special character?

This is not a bad idea, however, it needs to be documented with a comment in the code, as well as in the system preference. I think it would actually be better as its own system preference.

Nick, I'm aware of your 'hack'--now, I'm unable to verify if quickly, but as far as I remember it didn't solve this case. If it did, there would be no point for me to dig into this.

If I remember well, the patch intends to treat the question mark, in the function _query_regex_escape_process, in exactly the same way as it is done with slash: instead of escaping/unescaping only '/': (?=/) it escapes also '?': (?=[/\?]). Maybe I could extend the existing comment, but I don't feel like this is a candidate for a new syspref...?

I think it would be nice if we could keep the truncation feature without requiring to escape it. It's easy to explain to people to use ?, but much harder to explain \? - also quite a change in behavior at this point maybe?

I was wondering, if ? replaces one or more characters... why does it not work as a wild card if there really is a question mark?

Well, IMHO it will be not so easy to explain to an ordinary user (who usually will not read complicated introductory texts, and what we are discussing is a very special case) how '?' functions with ES. Moreover, I expect that an ordinary user will rather use '?' as a verbatim '?' (from copy-paste) than as a wildcard. So, I expect (and have already experienced) alarms from the users who are not able to find a book (with a title containing '?', copied from other data source) in the local catalog even if it is there. If I understand well, in query_string ES query, '?' stands for a single character. But, at the same time, after a normal text analysis of a string containing a question mark attached at the end of the word, the '?' character is removed from what is stored in the index. So, the query 'are?' will not find an original string 'are?' but it would find for instance 'ares', 'arer', 'area' etc. (Similarly, '/' has a special meaning in ES but at the same time can be put unconsciously by an ordinary user as a part of query string -- and hence, if I get it right, the option QueryRegexEscapeOptions of escaping it.) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #12 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Katrin Fischer from comment #11)

? and * are quite common for truncation and databases and the like, people have been writing tutorials on Elasticsearch etc. like: https://drive.google.com/file/d/17-YVjG6IJKeDmDCKuF7ZYvcTfzDYyxwa/view That's also been promoted on the BywaterSolutions blog.

So next question would be, why * and ? are handled differently... could the behaviour of Elasticsearch removing the ending ? be changed?

The fundamental differences between ? and * are: 1. * very rarely would be present in ordinary users' copy-paste queries, 2. even if, * means '0 or more characters' (like /.*/ in Perl) -- so it would not create problems -- whereas ? means _exactly_ one character (like Perlish /./) -- and therefor is quite problematic in the context we are in. Still, I would not expect from the users of the catalogue the knowledge about quite advanced, wildcarded ES queries... s/he is usually is not interested in what is in the back-end. Hence, I would prefer to prepare Koha to first "understand" a normal user instead of advanced search engines operator. And still, the basic scenario is: the user does copy-paste search and does not get the result -- which for him/her (and therefor for me) is not acceptable. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #14 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Nick Clemens from comment #13)

If you change or add a title index with 245(ab) then the "hack" does work. In fact, it's probably a good idea because any quoted 'full' title search including a subtitle will fail otherwise

Thank you for this comment. Yes, reformulating the search field definition in mappings would be of some help (but which search field? 'title' for sure - into 245(ahbnpc) at least - but will it solve all issues?). But it would not solve the problem universally. For instance, having the title in the record: "$a Will libraries survive? : $b the future of public libraries" and querying: "Will libraries survive? future of public libraries" (without 'the' in this case, but in general: missing any element from inside the recorded element) will still give the user no results. And this is not just for teasing--often the author taken from another data source would be transcribed differently in 245 $c (like omitting 'by' for instance). -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 Galen Charlton <gmcharlt@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |gmcharlt@gmail.com --- Comment #16 from Galen Charlton <gmcharlt@gmail.com> --- Since this remains a definite usability issue, would it be sufficient to get a fix in place by adding a system preference? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #18 from Janusz Kaczmarek <januszop@gmail.com> --- I'm not sure if I would like to have one preference (even if in two incarnations: OPAC and staff) switching off all the ES wildcard and truncation special characters... From some months of experience with Koha + ES I would say, it is '/' and '?' (sometimes also '!') that are problematic. There is no problem with most useful '*' or '~' -- even if unconsciously used by a user.

From my perspective it would be best to have the special meaning of '?', '/', '!' switched off by default, while '*' and '~' enabled.

-- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #20 from David Cook <dcook@prosentient.com.au> --- Yeah this is a tough one. Sometimes I think that we shouldn't try to be too smart, and instead just be more honest with end users. Something like 'Tip: a question mark is a reserved character! To use it, you must escape it like this: title:"Who is there\?"' When we try to be too smart programmatically, we end up in nightmare situations. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #22 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- (In reply to Janusz Kaczmarek from comment #21)

OK, but how about the general idea like this: a "normal" search should take into account only letters and digits and require from the user no special knowledge about the underlying technology, query grammar, expert constructs etc. In this case a "copy and paste" search will always return some results if there are record matching the search keywords.

At the same time, in advanced search -- a checkbox to enable ES grammar (i.e. not to strip any punctuation, parentheses, quotes etc.).

Wouldn't it be what an average inexperienced user would expect from a search tool? What do you think?

I like the idea of a checkbox or option to enable use of special syntax - if escaping by default I would want some sort of 'i'/info marker next to the search box just so people know -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #24 from David Cook <dcook@prosentient.com.au> --- (In reply to Nick Clemens (kidclamp) from comment #15)

I am not against the concept of auto-escaping the question mark, however, I do think it needs to be controlled by a syspref. Possibly with different options for staff and opac.

Agreed. When we're using the Advanced Search, I'd say we should auto-escape all reserved characters, since it's a structured search. Since we (rightly or wrongly) allow complex queries in the regular "Simple" Search, we're caught in a bit of a tough place. We're giving users the power to write complex queries... but we can't trust them to escape reserved characters either. It's a bit... absurd really. However, for many years, we've had this absurd approach, so for backwards compatibility I think we need to keep it going.

The code and the preference description also need to be clear about what they are doing - silently escaping '?' in the code that handles regex searching is confusing

Agreed. The splinter review looks identical at a glance. If I recall correctly, you can split regex across newlines. Alternatively, we might be able to build it from a series of strings. The current code isn't readable/maintainable. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #26 from David Cook <dcook@prosentient.com.au> --- (In reply to Janusz Kaczmarek from comment #12)

Still, I would not expect from the users of the catalogue the knowledge about quite advanced, wildcarded ES queries... s/he is usually is not interested in what is in the back-end. Hence, I would prefer to prepare Koha to first "understand" a normal user instead of advanced search engines operator.

One more comment and then I'll be done. (Apologies if my comments are coming across overly negative. I've spent over a decade looking at Koha's search code, and I've had a lot of librarians grumpy with me about Koha's search functionality in that time. More now with the switch to Elasticsearch than ever really...) The difficulty here is that we're already expecting them to understand OR/AND and () when it comes to Elasticsearch. One could argue that we should escape () and wrap every query in double quotes, if we don't think a normal user can understand complex search queries. But that will really irritate power users (on both staff interface and OPAC). And that's not what we've done historically. We try to outsmart the user, and it only kind of works. Sometimes I think we should just make a full decision one way or another. The Google style search box should probably just be a simple search with auto-escaping for reserved characters. Then we could have a Power Search or something like that in the Advanced Search area for power users where we "take off the kid gloves". If you use Google as an example, they actually do explain their search query syntax on their Advanced Search page. (Of course, I just found a little bug in Google's Advanced Search. Nothing is perfect...) I think that's where system preferences can be powerful, and individual libraries could then also analyze the search behaviour of users to see how they should tweak their search settings. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 carthur@slolibrary.org <carthur@slolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |carthur@slolibrary.org -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #147161|0 |1 is obsolete| | --- Comment #27 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 194458 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=194458&action=edit Bug 28884: Add 'ElasticsearchEscapeCharacters' and 'OpacElasticsearchEscapeCharacters' system preferences -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #29 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 194460 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=194460&action=edit Bug 28884: Unit tests -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #31 from Janusz Kaczmarek <januszop@gmail.com> --- I am returning to this old thread, which still seems to be unresolved. I propose a solution based on system preferences, separate for OPAC and for staff. Thank you for your comments. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Comma delimited| |Pontificia Università di list of Sponsors| |San Tommaso d'Aquino | |(Angelicum) Sponsorship status|--- |Sponsored -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 Roman Dolny <roman.dolny@jezuici.pl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #194459|0 |1 is obsolete| | Attachment #194460|0 |1 is obsolete| | Attachment #194461|0 |1 is obsolete| | --- Comment #32 from Roman Dolny <roman.dolny@jezuici.pl> --- Created attachment 195025 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=195025&action=edit Bug 28884: ElasticSearch: Question mark in title search returns no results Test plan: ========== 1. Have a ktd instance with Elasticsearch enabled. 2. In standard ktd installation there is a record with a title "Typical girls? : the story of the Slits." (with question mark attached to a word) - biblio # 229. We will try to find it. 3. Go to advanced search (OPAC or intranet) and try a *title* search with: Typical girls? the story of the Slits You should get no results. From the main page try a ccl-style search: ti:Typical girls? the story of the Slits You should get no results too. 4. Apply the patch ; updatedatabase ; restart_all. 5. Put ? in OpacElasticsearchEscapeCharacters and/or ElasticsearchEscapeCharacters system preference. 4. Repeat the searches from p. 3. You should get the record if a system preference was set for the given interface. Sponsored-by: Pontificia Università di San Tommaso d'Aquino (Angelicum) Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #34 from Roman Dolny <roman.dolny@jezuici.pl> --- Created attachment 195027 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=195027&action=edit Bug 28884: Add 'ElasticsearchEscapeCharacters' and 'OpacElasticsearchEscapeCharacters' system preferences Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #195025|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #195027|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #36 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 196112 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=196112&action=edit Bug 28884: Unit tests Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #38 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- Created attachment 196114 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=196114&action=edit Bug 28884: (QA follow-up) Add some missing pod Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |26.05.00 released in| | Status|Passed QA |Pushed to main -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 Miranda Nero <mnero@oslri.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mnero@oslri.net -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28884 --- Comment #41 from Jacob O'Mara <jacob.omara@openfifth.co.uk> --- Thanks all, pushed to 25.11.x -- You are receiving this mail because: You are watching all bug changes.