https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |ere.maijala@helsinki.fi |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #1 from Ere Maijala <ere.maijala@helsinki.fi> --- Created attachment 78889 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=78889&action=edit Bug 21357: Add elision filtering to default ES index config To test: 1. Rebuild Elasticsearch index with the new config 2. Add a record with "l'avion" in the title 3. Verify that the record can be found with both "l'avion" and "avion" -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nicolas.legrand@bulac.fr -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #4 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I think a lot of libraries have French materials, is it specific to French catalouging or just to data? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #6 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- (In reply to Ere Maijala from comment #5)

If it's decided this shouldn't be part of the default rules, we could perhaps still include these as a commented-out example, right?

I'd prefer to propose an entire french-specific file. I attach the one I'm working on. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #8 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Fridolin SOMERS from comment #6)

(In reply to Ere Maijala from comment #5)

...

If it's decided this shouldn't be part of the default rules, we could perhaps still include these as a commented-out example, right?

I'd prefer to propose an entire french-specific file. I attach the one I'm working on.

Are we sure those rules would not be helpful for other installations as well? I can understand if it's specific to UNIMARC, but otherwise it might be more helpful to develop good defaults together. At the moment we have the problem with Zebra and the sort files - some language specific sort files are really good and others have only minimal information. If you have a library with multiple languages (we do) one file that fits multiple would be better. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #10 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I think ideally things shoudl be as easy as possible. I think all of our libraries have materials in various languages - the Goethe institute libraires being a prominent example. I don't want to run each of them with a different Elasticsearch configuration if possible. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #12 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Serious question: Why would we want stop words? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #14 from Nicolas Legrand <nicolas.legrand@bulac.fr> --- Ay up friends, We are a language library and we have more than 350 different languages and lots of writing systems. Our interface is only in French (the language of our country) or in English (the science's new lingua franca). I we had installed other languages, we should have installed all languages available, which even today seems overkill, while everyone seems OK with French and English. Therefore, having stemming based on the interface language doesn't look a good option for us. OK maybe we're weird, nonetheless I think most University libraries have books in a lot of different languages and not just one or two. With our context, stemming itself seems quite trying. We aren't yet eager about it. So before having the full stemmed engine picky about which language is what, what have done Ere (thank you !) will certainly suit us. If it is not generic enough to be in Koha... Well we will hack it on our instance. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #16 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Ok I will propose the stemming configuration on wiki page. This patch is OK, but for me the "apostrophe" filter is not needed. I've checked with analyse API on ES : curl -X GET 'http://localhost:9200/koha_master_biblios/_analyze?pretty=yes&analyzer=analyser_phrase' -d "dans l'avion" With patch it returns : "token" : "dans lavion" Without patch it returns : "token" : "dans l'avion" Checking with only "l'avion" both with and without patch returns "avion". -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #17 from Ere Maijala <ere.maijala@helsinki.fi> --- The apostrophe filter is for the other cases where an apostrophe is not a part of an elision. It's not stripped in the char filter like other punctuation so that the elision filter can do its job but needs to be done afterwards. Try analysis for e.g. "Merriam-Webster's Collegiate Dictionary". I agree that this bug should only be about elisions. Stop words are another thing completely, and there are strong reasons for not using a stop word list like Katrin described. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff --- Comment #19 from Ere Maijala <ere.maijala@helsinki.fi> --- I'd like to either move this forward. I think elision handling is important even if the language used does not use them since they could be used in names etc. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Marjorie Barry-Vila <marjorie.barry-vila@collecto.ca> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |marjorie.barry-vila@collect | |o.ca -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Björn Nylén <bjorn.nylen@ub.lu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #79375|0 |1 is obsolete| | --- Comment #20 from Björn Nylén <bjorn.nylen@ub.lu.se> --- Created attachment 86670 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=86670&action=edit Bug 21357: Add elision filtering to default ES index config To test: 1. Rebuild Elasticsearch index with the new config 2. Add a record with "l'avion" in the title 3. Verify that the record can be found with both "l'avion" and "avion" Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr> Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #21 from Björn Nylén <bjorn.nylen@ub.lu.se> --- We signed off this bug as we think it's a good feature as is. We just recently had a coworker noting this behaviour. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #24 from Ere Maijala <ere.maijala@helsinki.fi> --- Created attachment 87902 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=87902&action=edit Bug 21357: Add uppercase articles to the elision filter -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #26 from Séverine Queune <severine.queune@bulac.fr> --- Hi Ere, Everything works fine for the single letters + apostrophe for both upper and lower case, but it doesn't work for the double letters QU : here the title of my notice : Qu'ils me laissent tranquille ! If I search : - QU'ILS --> failed - Qu'ils --> ok - qu'ils --> failed (same list of result I had for QU'ILS) - qU'ILS --> ok I created another exemple and had no problem due to uper or lower case : - Aujourd'hui --> ok - AUJOURD'HUI --> ok - AUJOURd'HUI --> ok As the 'D' and 'd' is on your list, could my first exemple behave correctly if you add only 'U' and 'u' on the settings ? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #28 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- At last I'm going to work on this again. I understand way more Elasticsearch now :D I know now that we can use french elision on a non-french catalog, even with an unwanted impact the ranking will give priority to correct match. So we can provide this in default configuration. Thanks to all for your existing participation on this bug. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #30 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Ohhh playing with ES API with explain analyze : https://www.elastic.co/guide/en/elasticsearch/reference/current/_explain_ana... I see that elision is only with the begining of the token. It explains a lot on analyzer_phrase (comment 16) I continue my tests. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com --- Comment #32 from Julian Maurice <julian.maurice@biblibre.com> --- I think that enabling french analyzer (or parts of french analyzer) by default is not the best solution. Sure it will work with french catalogs, but what will happen when one want to add support for another language (or, for the weird people above, 350 other languages! :)) ? I think a better solution is to use multi-fields with multiple analyzers, as suggested here : https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields... The idea is to have multiple "subfields" for each field, one subfield per language. For instance: - `title` will be language agnostic - `title.lang_en` will use an english analyzer - `title.lang_fr` will use a french analyzer When searching, Koha can search in all these fields, and the query will be analyzed by all analyzers. I've been testing this idea this week, and I made a small patch that I will submit here. It is only a proof of concept for now, but I hope it will help -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #34 from Julian Maurice <julian.maurice@biblibre.com> --- Changing status to needs signoff to get opinions -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #36 from Ere Maijala <ere.maijala@helsinki.fi> --- I can't really see the benefit since, as far as I can see, elision handling is not prone to cause conflicts with other language analysis. Separating analysis for different languages also won't work for mixed-language fields. Think about names and a (very fictional) example phrase "Images from movie l'Avion". You'd get either elision filtering or English stemming but not both. For sure it will still be found with a simple keyword search, but it breaks at least adjacent word searches and relevance ranking. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #38 from Ere Maijala <ere.maijala@helsinki.fi> --- (In reply to Julian Maurice from comment #37)

Ellision might not cause troubles (but what about names like "D'Amato" ?). I'm thinking about the next step : stemming is very different from one language to another and we need to find a way to have stemming for multi-language catalogs.

I agree that stemming is difficult, and I'e tried to purposefully limit this bug to elisions.

...

Separating analysis for different languages also won't work for mixed-language fields. Think about names and a (very fictional) example phrase "Images from movie l'Avion". You'd get either elision filtering or English stemming but not both. For sure it will still be found with a simple keyword search, but it breaks at least adjacent word searches and relevance ranking.

Nothing would work perfectly with mixed-language fields. But in this particular example, you could have another subfield `lang_en_fr` that does english stemming and french elision

And maybe lang_fi_fr, lang_sv_fr etc. This gets complicated pretty quickly. And I'm afraid separating the different language analysis chains doesn't solve the issue with stemming etc. because you'd need to avoid indexing into "wrong" fields which would require you to know what language the string to be indexed is in. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #40 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Julian Maurice from comment #39) What about using a record field to determine the language of the record and applying different analyser settings based on that? 008/35-37 or 041 in marc21, I don't know if there is a UNIMARC equivalent -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #42 from Ere Maijala <ere.maijala@helsinki.fi> --- At least with MARC 21 that's the language of the catalogued item, not the language of the metadata record, right? We've been through this with our discovery interface and had to give up trying to guess the language. Additionally, a record can contain metadata fields in multiple languages. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #44 from Nicolas Legrand <nicolas.legrand@bulac.fr> --- Julian, for non latin scripts we are doing a latin transliteration and so are suppose to do all the ABES partners in France. Look at https://catalogue.bulac.fr/cgi-bin/koha/opac-detail.pl?biblionumber=976426 http://www.sudoc.fr/18117295X If I check the language here, it's Hebrew. I've got a 200$a with a Hebrew title in hebrew and another one with a Hebrew title in Latin alphabet. To know what to do, I have to look at 200$6 and 200$7 which are ABES non standard ways to describe non latin scripts (I like to think we are not using UNIMARC but ABESMARC). http://documentation.abes.fr/sudoc/regles/Catalogage/Regles_Multiecritures.h... Voilà. Don't know what to think yet, but now you know. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Michal Denar <black23@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #87901|0 |1 is obsolete| | Attachment #90191|0 |1 is obsolete| | Attachment #98169|0 |1 is obsolete| | --- Comment #45 from Michal Denar <black23@gmail.com> --- Created attachment 98244 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=98244&action=edit Bug 21357: Add elision filtering to default ES index config To test: 1. Rebuild Elasticsearch index with the new config 2. Add a record with "l'avion" in the title 3. Verify that the record can be found with both "l'avion" and "avion" Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr> Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se> Signed-off-by: Michal Denar <black23@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #47 from Michal Denar <black23@gmail.com> --- Created attachment 98246 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=98246&action=edit Bug 21357: Allow to use multiple ES language analyzers Test plan: 1. Apply this patch 2. If you have a custom `field_config.yaml` report the diff in your custom config 3. Create biblios in english and french 4. misc/search_tools/rebuild_elasticsearch.pl 5. Test biblio search at intranet with stemming and ellision in mind Examples of test queries: - "journal actualité" => should return results with "Journaux d'actualités" in it - "lord lady" => should return results with "Lords and Ladies" in it Signed-off-by: Michal Denar <black23@gmail.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Bouzid Fergani <bouzid.fergani@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #98244|0 |1 is obsolete| | --- Comment #48 from Bouzid Fergani <bouzid.fergani@inlibro.com> --- Created attachment 101305 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=101305&action=edit Bug 21357: Add elision filtering to default ES index config To test: 1. Rebuild Elasticsearch index with the new config 2. Add a record with "l'avion" in the title 3. Verify that the record can be found with both "l'avion" and "avion" Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr> Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se> Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Bouzid Fergani <bouzid.fergani@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #98246|0 |1 is obsolete| | --- Comment #50 from Bouzid Fergani <bouzid.fergani@inlibro.com> --- Created attachment 101307 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=101307&action=edit Bug 21357: Allow to use multiple ES language analyzers Test plan: 1. Apply this patch 2. If you have a custom `field_config.yaml` report the diff in your custom config 3. Create biblios in english and french 4. misc/search_tools/rebuild_elasticsearch.pl 5. Test biblio search at intranet with stemming and ellision in mind Examples of test queries: - "journal actualité" => should return results with "Journaux d'actualités" in it - "lord lady" => should return results with "Lords and Ladies" in it Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #52 from Bouzid Fergani <bouzid.fergani@inlibro.com> --- Created attachment 101310 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=101310&action=edit Bug 19482 - Add support for defining 'mandatory' mappings To test: 1 - Apply patch 2 - ./installer/data/mysql/updatedatabase.pl 3 - Reset ES mapping: Administration->Search engine configuration , button at bottom of page 4 - 'issues' and 'title' mapping under 'search fields' should be mandatory and not editable 5 - On 'Bibliographic records' tab you should not be able to delete the single entry for issues 6 - You should be able to delete 'title' mappings, however, at the final one you should be stopped by javascript 7 - Bonus: force remove the last mapping from the page using developer tools - attempt to save and should be warned of missing mandatory mapping Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Bouzid Fergani <bouzid.fergani@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bouzid.fergani@inlibro.com Attachment #101309|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Bouzid Fergani <bouzid.fergani@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #101311|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #55 from Ere Maijala <ere.maijala@helsinki.fi> --- Indeed my proposal was just to add elision filtering to the default ES index config without any code changes. The first patch still does that, but since then there have been additional ideas incorporated. I think that it would make sense to create separate bugs for them. Then it would only leave us with the question of whether we include elision filtering in the default configuration or just document this as a possibility. Either way is fine for me since in Finland we'll need to customize the default config regardless. I'm not sure how to reach consensus on this. I believe including filtering by default would make sense regardless of whether the catalog contains French titles since it helps searching e.g. author names too. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #57 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Katrin Fischer from comment #56)

+1 for default filtering. It's how it works now in the default and we haven't seen questions or complaints about this. To me it appears to be the expected behaviour.

And welcome back, Ere :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com --- Comment #59 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- So, where are we at with this one? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #61 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Ere Maijala from comment #60)

I think the last patch "Allow to use multiple ES language analyzers" should be a separate issue. The fix for case-insensitivity in the second patch is valid and necessary. I failed to even test without, and the Elasticsearch docs seem to have the meaning of true/false for the articles_case option reversed.

So could we get this back into testing with the first two patches? They still apply, just the third is conflicting. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff --- Comment #62 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I am making a brave step here (please don't kill me :) ) and putting this back in the test queue. I am not sure how to phrase a new bug for the now obsoleted 3rd patch, maybe someone could help? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 jmbroust <jean-manuel.broust@univ-lyon2.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jean-manuel.broust@univ-lyo | |n2.fr -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Simon Hohl <simon.hohl@dainst.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |simon.hohl@dainst.org --- Comment #64 from Simon Hohl <simon.hohl@dainst.org> --- In order to keep this bug alive: Our institution has several libraries in different countries requiring all kinds of char sets and punctuation supported, so we are also very much interested to have a generic solution to the problem. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #66 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- What would be the easiest option here to make everyone happy for now: Could it be made a configuration option somehow? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=14542 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #69 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Ere Maijala from comment #68)

(In reply to Katrin Fischer from comment #67)

...

(In reply to Katrin Fischer from comment #66)

...

What would be the easiest option here to make everyone happy for now: Could it be made a configuration option somehow?

I just found bug 14542 which seems to solve this for Zebra with ICU - could we not do the same for Elasticsearch?

Sure we could, if that's what makes everyone happy.

Sometimes making everyone happy is hard, but I feel that there is a need to solve this and we haven't moved an inch for some time. Looking for a next step forward. I feel for us the "Zebra" style solution would work. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #71 from Nicolas Legrand <nicolas.legrand@bulac.fr> --- I was very happy with Ere first proposal. Bug 14542 looks weirder to me. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |januszop@gmail.com --- Comment #73 from Janusz Kaczmarek <januszop@gmail.com> --- It is a pity that after almost five years this issue is still not solved ;) I take the opportunity to add some initial elisions from an Italian/international catalogue -- because the problem is not really French-specific: dell' o' nell' dall' all' un' sull' sant' ch' --to list only the most common. BTW, the problem of apostrophe is not only the problem of the initial elisions--also the saxon genetive creates problems: looking for "Shakespeare's" with "Shakespeare" will give no results unless QueryAutoTruncate is on. So maybe in analyzer_standard, as a third filter the apostrophe filter should be used as well? (which probably would also have some drawbacks...) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #75 from David Cook <dcook@prosentient.com.au> --- (In reply to Janusz Kaczmarek from comment #74)

... I mean the original apostrophe filter: https://www.elastic.co/guide/en/elasticsearch/reference/8.7/analysis- apostrophe-tokenfilter.html

According to the documentation, that was originally created for the Turkish language. In English, it would work well with apostrophe S ('s) but not so well for contractions like "don't" or "can't". It would also break languages like Ukrainian: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153#c14 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #77 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #76)

I am not sure I follow that argument. We already have been ignoring ' with Zebra for many years and never got a bug filed about it. On the other hand, we have a lot of people now that ask for it to ignore the '.

We don't ignore the apostrophe/single quote in Zebra. We change it to a space. That's significantly different than replacing it with nothing from a tokenization perspective.

Would it be possible to make it configurable? If not, I'd say we should just go with it. It's a big issue for French. And searching 'dont' is what I feel is more common than 'don t'.

I think I've talked about some options on a different bug about indexing both options, so there probably are some options (like indexing with and without punctuation) for making a Koha configuration for it. But it would probably be a bit error-prone as we'd need to change all of Koha to support it. I'm not using Elasticsearch yet, so I'm happy for people to try things out. Just trying to think of any pitfalls. And also hoping that we can try to keep Zebra and Elasticsearch similar in their configuration - or else we should think about deprecating Zebra and just focusing on Elasticsearch. Regarding "don't", I would hope that people would search for "don't" instead of "dont" or "don t". Note that "dont" is also a French word, so searches for "don't" would match for titles including the word "dont". But multilingual indexing is hard anyway... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 --- Comment #79 from David Cook <dcook@prosentient.com.au> --- I have a library catalogue with French, English, and Arabic records. We're using Zebra with ICU indexing, but it would be interesting to try out Elasticsearch on that catalogue, especially with these patches. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=36678 -- You are receiving this mail because: You are watching all bug changes.