https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 Phil Ringnalda <phil@chetcolibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |phil@chetcolibrary.org -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 --- Comment #1 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 164633 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=164633&action=edit Bug 33407: With ES and QueryAutoTruncate on, a search containing ISBD punctuation returns no results With ES and QueryAutoTruncate on, a search with punctuation surrounded by spaces (like ISBD punctuation copied from other catalogue or bibliography) returns no results. E.g.: in a search for "Maria Stuart ; Die Jungfrau von Orleans / Schiller" (coded in record as 245 10 $a Maria Stuart ; $b Die Jungfrau von Orleans / $c Schiller) -- both semicolon and slash cause problems. One had to remove them manually to get results which is not what is desired. Test plan ========= 1. Use ktd with Elasticsearch and ktd's test data (http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=1): 2. Make a search for the first ISBD zone taken verbatim from the first record: E Street shuffle : the glory days of Bruce Springsteen & the E Street Band / Clinton Heylin There should be no result. 3. Apply teh patch, restart plack. 4. Repeat the search. You should get the record (and onother one). Sponsored-by: Ignatianum University in Cracow -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |januszop@gmail.com |ity.org | CC| |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Change sponsored?|--- |Sponsored -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 Roman Dolny <roman.dolny@jezuici.pl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #164633|0 |1 is obsolete| | --- Comment #3 from Roman Dolny <roman.dolny@jezuici.pl> --- Created attachment 164957 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=164957&action=edit Bug 33407: With ES and QueryAutoTruncate on, a search containing ISBD punctuation returns no results With ES and QueryAutoTruncate on, a search with punctuation surrounded by spaces (like ISBD punctuation copied from other catalogue or bibliography) returns no results. E.g.: in a search for "Maria Stuart ; Die Jungfrau von Orleans / Schiller" (coded in record as 245 10 $a Maria Stuart ; $b Die Jungfrau von Orleans / $c Schiller) -- both semicolon and slash cause problems. One had to remove them manually to get results which is not what is desired. Test plan ========= 1. Use ktd with Elasticsearch and ktd's test data (http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=1): 2. Make a search for the first ISBD zone taken verbatim from the first record: E Street shuffle : the glory days of Bruce Springsteen & the E Street Band / Clinton Heylin There should be no result. 3. Apply teh patch, restart plack. 4. Repeat the search. You should get the record (and onother one). Sponsored-by: Ignatianum University in Cracow Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Patch doesn't apply CC| |domm@plix.at --- Comment #5 from Thomas Klausner <domm@plix.at> --- The patch does not apply anymore because the number of tests seem to have changed. The fix should be trivial. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 --- Comment #7 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Thomas Klausner from comment #5)

The patch does not apply anymore because the number of tests seem to have changed.

The fix should be trivial.

Thanks, Thomas. Please, try it now. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #167410|0 |1 is obsolete| | --- Comment #9 from Thomas Klausner <domm@plix.at> --- Created attachment 167420 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=167420&action=edit Bug 33407: Unit tests -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 Thomas Klausner <domm@plix.at> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #10 from Thomas Klausner <domm@plix.at> --- Works as advertised. I took a quick look at the added regex and .. wow. An escaped escaped slash :-) But it works, so: PASS QA! (my first!) PS: I removed a spurious newline in the code patch, and therefore had to re-upload both patches. I hope this is OK... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |24.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #13 from David Cook <dcook@prosentient.com.au> --- I think that this might break searches where people are relying on the punctuation to be included. Feels a bit like we're repeating mistakes we made with Zebra in the past... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- (In reply to Nick Clemens (kidclamp) from comment #14)

This is only removing punctuation surrounded by spaces, if the terms are quoted, it won't be removed.

We don't have unit tests that confirm that. Only one test was added for one particular scenario. Note that this query string isn't tested without QueryAutoTruncate in the unit tests...

It is difficult here to balance between exact results and meeting user expectations. I worry we are getting a bit confusing as we massage the queries more, but we still have user complaints about the over exactness of some features in ES

I am concerned about the massaging of queries. Could you give some examples of the over exactness? I have found it interesting needing to do 'local-number:"1"' instead of 'local-number:1' but I've adapted.

Looking over this one again, it seems the problem only does occur when QueryAutoTruncate is enabled - so possibly the regex should be surrounded by a checked for this pref? That would minimize the scope of the 'magic' here

It looks like the $truncate variable there is set by QueryAutoTruncate. But since QueryAutoTruncate is on out of the box, we can think of this as default behaviour. It can be tough to change things away from defaults. -- Our main issue with Elasticsearch is timeouts/inconsistent indexing and not a lack of compatibility in a number of areas (like date searching). I can't say wildcards/truncation has really been an issue. It's interesting though. Much left to investigate and improve... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |additional_work_needed --- Comment #17 from Nick Clemens (kidclamp) <nick@bywatersolutions.com> --- (In reply to David Cook from comment #15)

(In reply to Nick Clemens (kidclamp) from comment #14)

...

This is only removing punctuation surrounded by spaces, if the terms are quoted, it won't be removed.

We don't have unit tests that confirm that. Only one test was added for one particular scenario.

Note that this query string isn't tested without QueryAutoTruncate in the unit tests...

I added more tests for when truncate is enabled - it would be good to repeat the tests without the syspref - can we get a follow-up Janusz?

Could you give some examples of the over exactness? I have found it interesting needing to do 'local-number:"1"' instead of 'local-number:1' but I've adapted.

The above should be fixed by bug 32707. In general, Zebra was more forgiving: without truncation 'idea' would match 'ideas' or 'ideation' 'blues' would match 'blue' Punctuation too: dont/don't matched both ways

It looks like the $truncate variable there is set by QueryAutoTruncate.

But since QueryAutoTruncate is on out of the box, we can think of this as default behaviour. It can be tough to change things away from defaults.

Oops, I forgot the conditional, thanks :-)

Our main issue with Elasticsearch is timeouts/inconsistent indexing and not a lack of compatibility in a number of areas (like date searching). I can't say wildcards/truncation has really been an issue. It's interesting though. Much left to investigate and improve...

Hmm...we've done a number of enhancements to the indexer to improve this and ensure smaller batches to avoid timeouts - could you open a bug with more details? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 --- Comment #19 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #18)

(In reply to Nick Clemens (kidclamp) from comment #17)

...

In general, Zebra was more forgiving: without truncation 'idea' would match 'ideas' or 'ideation' 'blues' would match 'blue'

That would be due to the QueryStemming syspref I think.

Actually, even with the stemming, that would've still needed truncation. I'm a bit skeptical about this one, but open to hearing more. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com --- Comment #21 from Lucas Gass <lucas@bywatersolutions.com> --- Not going to backport this one until the follow-ups/additional work are QA'ed. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 --- Comment #22 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- @Janusz can you please have a look at Nick's follow-up tests? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 --- Comment #24 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Janusz Kaczmarek from comment #23)

(In reply to Katrin Fischer from comment #22)

...

@Janusz can you please have a look at Nick's follow-up tests?

I am sorry if it was not clear enough, but see Bug 37430 of July 2024--which was the answer to the Nick's suggestion (more comments, tiny adjustment, and expanded tests).

Hi Janusz, I was wondering about the proposed follow-up "Expand tests" as this is not in main yet. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 --- Comment #26 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Katrin Fischer from comment #25)

The last follow up no longer applies or was already applied? Please check.

If I recall it properly, Bug 37430 (cf. https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37430#c0) was a amended version, prepared after this one was already pushed to main. Nick's expanded test is there. So no worry. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33407 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to main |Needs documenting CC| |fridolin.somers@biblibre.co | |m --- Comment #27 from Fridolin Somers <fridolin.somers@biblibre.com> --- Looks like we can close -- You are receiving this mail because: You are watching all bug changes.