[Bug 27153] New: ElasticSearch should search keywords apostrophe blind
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Bug ID: 27153 Summary: ElasticSearch should search keywords apostrophe blind Change sponsored?: --- Product: Koha Version: 20.11 Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: jonathan.field@ptfs-europe.com We are often asked why keyword searches don't ignore apostrophes. If you searched for "carries war" on Google, the first hits that come up are for Carrie's War by Nina Bawden. If you do a keyword search in Koha for "carries war" you get 0 hits. If you search for "carrie's war" you will find copies of this book. Koha with ElasticSearch should search apostrophe-blind (ignore the apostrophe). This was a perennial problem with zebra. In zebra you had to change words-icu.xml to ignore apostrophes. We should deliver an ES index_config.yaml that handles this scenario. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |DUPLICATE --- Comment #1 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Jonathan, I believe this to be a duplicate of bug 21357 - maybe you could add your voice to that bug for a simplified solution (ignore ') *** This bug has been marked as a duplicate of bug 21357 *** -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #2 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 114234 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=114234&action=edit Bug 27153: Default to apostrophe blind search Keyword searching in elasticsearch should be apostrophe blind; this patch updates the default configuration to filter apostrophe's. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Resolution|DUPLICATE |--- Status|RESOLVED |REOPENED --- Comment #3 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- In my understanding elisions and apostrophes are not equal and as such distinct bugs can/should be opened to handle the two cases. This case is for apostrophes only and leaves elisions, which appear to be more difficult to reach a consensus, to the other bug. As such, I don't think this is a duplicate at all.. I've uploaded Jonathans patch. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |jonathan.field@ptfs-europe. |ity.org |com -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|20.11 |master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Abbey Holt <aholt@dubuque.lib.ia.us> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aholt@dubuque.lib.ia.us -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #4 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- I don't mind a simple solution, but can you explain the difference? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #5 from Nick Clemens <nick@bywatersolutions.com> --- With this simple apostrophe filter: Dont let the pigeon drive the batmobile will find: Don't let the pigeon drive the batmobile but, you get no results for: ti:"Don't let the pigeon drive the batmobile" I like the idea of a simple solution, but I am not sure it works for everyone. While elisions are different than contractions I believe they both use an apostrophe, so this will affect both. We are planning on testing the 'english' analyzer with one of our partners to help with this issue, I suspect that this may not have a one-size-fits-all solution - but will require the user to decide on which configuration works best in their library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #6 from Fridolin Somers <fridolin.somers@biblibre.com> --- (In reply to Katrin Fischer from comment #4)
I don't mind a simple solution, but can you explain the difference?
Elision removes the text : l'europe = europe This apostrophe filter creates : l'europe = leurope Wrong in french but surely ranking will bet better for the exact match. I have to test combined with ellision. But : Maybe it is bad for authorities : "Pierre d'Anjou" and Pierre Danjou". -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #7 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Could we treat it ' as a space and 'no space' at the same time maybe? And how do we currently handle this with Zebra? I guess we'd want Don't Dont don t Pierre d'Anjou Pierre dAnjou Pierre d Anjou to be the same? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Barbara Johnson <barbara.johnson@bedfordtx.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |barbara.johnson@bedfordtx.g | |ov --- Comment #8 from Barbara Johnson <barbara.johnson@bedfordtx.gov> --- This is a constant problem for our patrons who are used to searching Amazon or Google without having to worry about apostrophes. Fixing this issue would really improve the ability for our patrons to find items in our catalog. The status of this bug is listed as Needs Signoff. Is there a test plan? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #9 from Fridolin Somers <fridolin.somers@biblibre.com> --- Actually Zebra as a rule in words-icu.xml : <transliterate rule="\'>\ "/> But it may not be optimal to do the exact same thing in Elasticsearch. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #10 from David Cook <dcook@prosentient.com.au> --- It might be a good idea to harmonize the normalization used between Zebra and Elasticsearch just for consistency... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |In Discussion --- Comment #11 from Fridolin Somers <fridolin.somers@biblibre.com> --- (In reply to David Cook from comment #10)
It might be a good idea to harmonize the normalization used between Zebra and Elasticsearch just for consistency...
+1 So we just need to change replacement with a space character ? This will require a unit test I bet. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Janet McGowan <janet.mcgowan@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |janet.mcgowan@ptfs-europe.c | |om -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Jennifer Millikan <jmillikan@sja1840.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jmillikan@sja1840.org --- Comment #12 from Jennifer Millikan <jmillikan@sja1840.org> --- I am brand new to Koha, but I was told this bug is why I have to include apostrophes while searching. I tried searching for Don't Hate the Player title and came up with zero results. I did not use the apostrophe because I have never had to before in multiple catalog systems. I repeated the search with the apostrophe and found the title. I agree with this and think the searching should work both ways with or without the apostrophe. Thank you. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #13 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 139969 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=139969&action=edit Bug 27153: POC This is a small example to show that we can add an extra field to our default indexing that removes punctuation. If we then add that field to the fields we push, we can sarch both the regular and the punctuation stripped version To complete this we would need to add a 'search variations' field to the ES config page, for the user to decide which fields should be searched in all variations, and which shouldn't Or we could add this to all 'simple searches' from the search bar, and only search exact on advanced searches. I am happy to explore either option -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #14 from David Cook <dcook@prosentient.com.au> --- I had a librarian (from an English-only library) asking for this the other day, and I was wondering why we have apostrophes replaced with a space (both for CHR and ICU indexing with Zebra). But Frido's example makes a lot of sense: (In reply to Fridolin Somers from comment #6)
Elision removes the text : l'europe = europe This apostrophe filter creates : l'europe = leurope
Wrong in french but surely ranking will bet better for the exact match.
And in English, we'd have "father's" become "fathers", which means "father" wouldn't match unless you're using right truncation (which Koha typically does out of the box I suppose). But then there's a Ukrainian word під'їзд which if you broke it into під їзд, you'd get hits for під which is a totally unrelated word. -- I know people have praised Google here, but it's not perfect either. Try searching "l'arbre під'їзд" and try searching "під'їзд l'arbre", and you'll get wildly different results. It seems that Google tries to determine the language of the search query (possibly based off the first word), and then analyze the search string based off that. For "l'arbre під'їзд", you get many results with "L'Arbre", "des arbres", "arbre", etc. Most results seem to include Russian rather than Ukrainian. But for "під'їзд l'arbre", you only get a few results that have the string "L'Arbre"and most of the results are Ukrainian. So Google is likely determining the language of the search string and then applying a language-specific analyzer. "My mom" in Chinese is 我的妈妈. 的 is the character that denotes possession of mom (妈妈) by me/I (我). You might think then you could replace 的 with a space to separate the 2 nouns... except 的 isn't always used that way. The word 目的 means "goal". Doing anything to 的 would compromise the word, unless you're able to understand the context that it's used in. If you google "我的妈妈", eventually it strips off "我的" and just searches for "妈妈" since that's the main noun in the phrase. Search is hard. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- (In reply to Katrin Fischer from comment #7)
Could we treat it ' as a space and 'no space' at the same time maybe?
Without language-specific analysis, this might be the best way to go... (In reply to Nick Clemens from comment #13)
Created attachment 139969 [details] [review] Bug 27153: POC
This is a small example to show that we can add an extra field to our default indexing that removes punctuation.
If we then add that field to the fields we push, we can sarch both the regular and the punctuation stripped version
I'm not very familiar with the Elasticsearch integration, but that POC doesn't touch the indexing, does it? Isn't it just modifying the search? (I don't know that it's touching apostrophes either?) If you searched "carries war" but it had been indexed as "carrie s war", then you still wouldn't get a hit? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #16 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to David Cook from comment #15)
(In reply to Katrin Fischer from comment #7)
Could we treat it ' as a space and 'no space' at the same time maybe? Without language-specific analysis, this might be the best way to go...
That's sort of what the POC does, but just for "as apostrophe" "as no space" - we could add a third field for converting apostrophes to spaces
(In reply to Nick Clemens from comment #13)
Created attachment 139969 [details] [review] [review] Bug 27153: POC I'm not very familiar with the Elasticsearch integration, but that POC doesn't touch the indexing, does it?
It does, it adds an extra field, with a punctuation filter to the indexes for 'default' fields - that filter removes all ASCII punctuation characters
Isn't it just modifying the search?
It then adds that field to the fields being searched in a 'keyword' search
If you searched "carries war" but it had been indexed as "carrie s war", then you still wouldn't get a hit? Not with this one, but "carrie's war" or "carries war" would work
The nice thing is, adding an extra index means we can generically search all versions of the field, but can also target specific ones: title.punc_removed: carries title.raw: carrie's So staff can be specific, while making standard search more friendly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #17 from Jonathan Field <jonathan.field@ptfs-europe.com> --- I think the POC is a good compromise. Difficult are this but we need to do something to address it as it comes up constantly. We're happy to go with this. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #18 from David Cook <dcook@prosentient.com.au> --- I tried to understand this without applying the patches, but it's too hard without the context, so let's see... I assume that the second patch "POC" is an alternative patch and not an additional patch... First patch: - adds apostrophe filter to "analyzer_standard" which is used for all default searches... - apostrophe strips out apostrophes Second patch: - Adds a "punc_removed" field (which uses the analyzer_stdno analyzer which already has a punctuation filter) to "default" under "search". If I understood Elasticsearch and Koha's integration better, I would probably understand this, but I don't currently. - Reading through Koha/SearchEngine/Elasticsearch.pm and https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields... it looks like "phrase", "raw", "ci_raw" and "punc_removed" are "fields" only used when queried specifically - In Koha/SearchEngine/Elasticsearch/QueryBuilder.pm adds title.punc_removed field to all Elastic queries. We do something similar in Zebra in C4::Search::_build_weighted_query() where we add title fields to search. I wonder a bit if adding title.punc_removed there will have unintended consequences but it seems similar to Zebra so might not be a big drama. Regarding Zebra, I don't think we can get full feature parity here. However, we could potentially add a Title-punc_removed index and update ./etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl to strip punctuation for it, and then add that into C4::Search::_build_weighted_query() (or elsewhere). -- I think the "POC" patch would need some testing to make sure there aren't any unintended consequences, but overall it sounds like a reasonable proposition. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=31695 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Dani Elder <danielle.elder@law.utexas.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |danielle.elder@law.utexas.e | |du -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #19 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #18)
Regarding Zebra, I don't think we can get full feature parity here. However, we could potentially add a Title-punc_removed index and update ./etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl to strip punctuation for it, and then add that into C4::Search::_build_weighted_query() (or elsewhere).
Alternatively, we could store a punctuation removed version into the same index. That way you end up indexing both "Carrie's war" and "Carries War". But then do you do that for all indexes? That would double the size of your index... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=21357 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #20 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 155145 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155145&action=edit Bug 27153: DB Updates We start by adding the field to the table It is possible a library would want two versions of the same field in the indexing. We need to make the filter part of the primary key in order to allow this -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #21 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 155146 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155146&action=edit Bug 27153: DO NOT PUSH: Schema updates -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #22 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 155147 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155147&action=edit Bug 27153: Interface updates This patch adds a new column to the 'Bibliographic records' tab in Administration > Search engine configuration (Elasticsearch) To test: 1 - Confirm the new 'filter' column shows 2 - Update an existing field to set filter to punctuation 3 - Confirm it can be saved 4 - Add a new field 5 - Confirm it saves correctly 6 - Unset filter for a field 7 - Confirm it saves -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #23 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 155148 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155148&action=edit Bug 27153: Add filter option to Elasticsearch indexing This patch uses the filter option from the mappings to add a value_callback to reord processing for indexing. Fields defined with 'punctuation' filter will have all punctuation stripped when conveted to documents. Tests are updated. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #24 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 155149 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155149&action=edit Bug 27153: Adjust mappings to add title without punctuation This patch finally adjust the default mappings to add a new field to the elastic index with the title stripped of punctuation This method optionally allows the library to place the filtered version in the same search field, or a new search field. The default mappings will include the filtered version in a keyword search, but not a targeted search To test: 1 - Import some affected records via z3950, e.g.: Carrie's war 1,000 Japanese words 2 - Search (using Elasticsearch) for the titles without including punctuation Carries war 1000 Japanese words 3 - No results 4 - Reload mappings and reindex perl misc/search_tools/rebuild_elasticsearch.pl -v -r 5 - Search again 6 - Success! 7 - Search title specifically: ti:Carries war 8 - No results 9 - Adjust mappings.yaml to add second mapping for 245 to title index: - facet: '' marc_field: 245abp marc_type: marc21 sort: 1 suggestible: 1 filter: punctuation 10 - Reload mappings and reindex 11 - Repeat 7 12 - Success -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|jonathan.field@ptfs-europe. |nick@bywatersolutions.com |com | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #114234|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #139969|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=34695 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #25 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 155230 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155230&action=edit Bug 27153: Adjust kohastructure for new primary key -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 ByWater Sandboxes <bws.sandboxes@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155230|0 |1 is obsolete| | --- Comment #26 from ByWater Sandboxes <bws.sandboxes@gmail.com> --- Created attachment 155502 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155502&action=edit Bug 27153: Adjust kohastructure for new primary key Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 ByWater Sandboxes <bws.sandboxes@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155146|0 |1 is obsolete| | --- Comment #27 from ByWater Sandboxes <bws.sandboxes@gmail.com> --- Created attachment 155508 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155508&action=edit Bug 27153: DO NOT PUSH: Schema updates Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 ByWater Sandboxes <bws.sandboxes@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155147|0 |1 is obsolete| | --- Comment #28 from ByWater Sandboxes <bws.sandboxes@gmail.com> --- Created attachment 155509 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155509&action=edit Bug 27153: Interface updates This patch adds a new column to the 'Bibliographic records' tab in Administration > Search engine configuration (Elasticsearch) To test: 1 - Confirm the new 'filter' column shows 2 - Update an existing field to set filter to punctuation 3 - Confirm it can be saved 4 - Add a new field 5 - Confirm it saves correctly 6 - Unset filter for a field 7 - Confirm it saves Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 ByWater Sandboxes <bws.sandboxes@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155148|0 |1 is obsolete| | --- Comment #29 from ByWater Sandboxes <bws.sandboxes@gmail.com> --- Created attachment 155510 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155510&action=edit Bug 27153: Add filter option to Elasticsearch indexing This patch uses the filter option from the mappings to add a value_callback to reord processing for indexing. Fields defined with 'punctuation' filter will have all punctuation stripped when conveted to documents. Tests are updated. Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 ByWater Sandboxes <bws.sandboxes@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155149|0 |1 is obsolete| | --- Comment #30 from ByWater Sandboxes <bws.sandboxes@gmail.com> --- Created attachment 155511 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155511&action=edit Bug 27153: Adjust mappings to add title without punctuation This patch finally adjust the default mappings to add a new field to the elastic index with the title stripped of punctuation This method optionally allows the library to place the filtered version in the same search field, or a new search field. The default mappings will include the filtered version in a keyword search, but not a targeted search To test: 1 - Import some affected records via z3950, e.g.: Carrie's war 1,000 Japanese words 2 - Search (using Elasticsearch) for the titles without including punctuation Carries war 1000 Japanese words 3 - No results 4 - Reload mappings and reindex perl misc/search_tools/rebuild_elasticsearch.pl -v -r 5 - Search again 6 - Success! 7 - Search title specifically: ti:Carries war 8 - No results 9 - Adjust mappings.yaml to add second mapping for 245 to title index: - facet: '' marc_field: 245abp marc_type: marc21 sort: 1 suggestible: 1 filter: punctuation 10 - Reload mappings and reindex 11 - Repeat 7 12 - Success Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 ByWater Sandboxes <bws.sandboxes@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155508|0 |1 is obsolete| | --- Comment #31 from ByWater Sandboxes <bws.sandboxes@gmail.com> --- Created attachment 155512 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155512&action=edit Bug 27153: DO NOT PUSH: Schema updates Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 ByWater Sandboxes <bws.sandboxes@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155502|0 |1 is obsolete| | --- Comment #32 from ByWater Sandboxes <bws.sandboxes@gmail.com> --- Created attachment 155513 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155513&action=edit Bug 27153: Adjust kohastructure for new primary key Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 ByWater Sandboxes <bws.sandboxes@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155509|0 |1 is obsolete| | --- Comment #33 from ByWater Sandboxes <bws.sandboxes@gmail.com> --- Created attachment 155514 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155514&action=edit Bug 27153: Interface updates This patch adds a new column to the 'Bibliographic records' tab in Administration > Search engine configuration (Elasticsearch) To test: 1 - Confirm the new 'filter' column shows 2 - Update an existing field to set filter to punctuation 3 - Confirm it can be saved 4 - Add a new field 5 - Confirm it saves correctly 6 - Unset filter for a field 7 - Confirm it saves Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 ByWater Sandboxes <bws.sandboxes@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155510|0 |1 is obsolete| | --- Comment #34 from ByWater Sandboxes <bws.sandboxes@gmail.com> --- Created attachment 155515 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155515&action=edit Bug 27153: Add filter option to Elasticsearch indexing This patch uses the filter option from the mappings to add a value_callback to reord processing for indexing. Fields defined with 'punctuation' filter will have all punctuation stripped when conveted to documents. Tests are updated. Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #35 from ByWater Sandboxes <bws.sandboxes@gmail.com> --- Created attachment 155516 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155516&action=edit Bug 27153: Adjust mappings to add title without punctuation This patch finally adjust the default mappings to add a new field to the elastic index with the title stripped of punctuation This method optionally allows the library to place the filtered version in the same search field, or a new search field. The default mappings will include the filtered version in a keyword search, but not a targeted search To test: 1 - Import some affected records via z3950, e.g.: Carrie's war 1,000 Japanese words 2 - Search (using Elasticsearch) for the titles without including punctuation Carries war 1000 Japanese words 3 - No results 4 - Reload mappings and reindex perl misc/search_tools/rebuild_elasticsearch.pl -v -r 5 - Search again 6 - Success! 7 - Search title specifically: ti:Carries war 8 - No results 9 - Adjust mappings.yaml to add second mapping for 245 to title index: - facet: '' marc_field: 245abp marc_type: marc21 sort: 1 suggestible: 1 filter: punctuation 10 - Reload mappings and reindex 11 - Repeat 7 12 - Success Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #36 from ByWater Sandboxes <bws.sandboxes@gmail.com> --- Created attachment 155517 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=155517&action=edit Bug 27153: Adjust kohastructure for new primary key Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155511|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155513|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Dani Elder <danielle.elder@law.utexas.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Michelle Spinney <mspinney@clamsnet.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mspinney@clamsnet.org -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com Keywords| |rel_23_11_candidate -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact| |martin.renvoize@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155145|0 |1 is obsolete| | --- Comment #37 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 157732 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157732&action=edit Bug 27153: DB Updates We start by adding the field to the table It is possible a library would want two versions of the same field in the indexing. We need to make the filter part of the primary key in order to allow this Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155512|0 |1 is obsolete| | --- Comment #38 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 157733 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157733&action=edit Bug 27153: DO NOT PUSH: Schema updates Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155514|0 |1 is obsolete| | --- Comment #39 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 157734 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157734&action=edit Bug 27153: Interface updates This patch adds a new column to the 'Bibliographic records' tab in Administration > Search engine configuration (Elasticsearch) To test: 1 - Confirm the new 'filter' column shows 2 - Update an existing field to set filter to punctuation 3 - Confirm it can be saved 4 - Add a new field 5 - Confirm it saves correctly 6 - Unset filter for a field 7 - Confirm it saves Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155515|0 |1 is obsolete| | --- Comment #40 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 157735 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157735&action=edit Bug 27153: Add filter option to Elasticsearch indexing This patch uses the filter option from the mappings to add a value_callback to reord processing for indexing. Fields defined with 'punctuation' filter will have all punctuation stripped when conveted to documents. Tests are updated. Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155516|0 |1 is obsolete| | --- Comment #41 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 157736 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157736&action=edit Bug 27153: Adjust mappings to add title without punctuation This patch finally adjust the default mappings to add a new field to the elastic index with the title stripped of punctuation This method optionally allows the library to place the filtered version in the same search field, or a new search field. The default mappings will include the filtered version in a keyword search, but not a targeted search To test: 1 - Import some affected records via z3950, e.g.: Carrie's war 1,000 Japanese words 2 - Search (using Elasticsearch) for the titles without including punctuation Carries war 1000 Japanese words 3 - No results 4 - Reload mappings and reindex perl misc/search_tools/rebuild_elasticsearch.pl -v -r 5 - Search again 6 - Success! 7 - Search title specifically: ti:Carries war 8 - No results 9 - Adjust mappings.yaml to add second mapping for 245 to title index: - facet: '' marc_field: 245abp marc_type: marc21 sort: 1 suggestible: 1 filter: punctuation 10 - Reload mappings and reindex 11 - Repeat 7 12 - Success Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #155517|0 |1 is obsolete| | --- Comment #42 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 157737 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=157737&action=edit Bug 27153: Adjust kohastructure for new primary key Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #43 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- This adds a nice flexible and configurable way of achieving the desired goal. No regressions found, the QA scripts are happy and unit tests are all passing. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |23.11.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #44 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 23.11. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |release-notes-needed Status|Pushed to master |Needs documenting --- Comment #45 from Fridolin Somers <fridolin.somers@biblibre.com> --- Enhancement not pushed to 23.05.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs documenting |Pushed to master Severity|enhancement |normal --- Comment #46 from Nick Clemens <nick@bywatersolutions.com> --- I marked as 'normal' now - while this adds new functionality I think it resolves a bug for users where records are not being found. Reconsider? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #47 from Fridolin Somers <fridolin.somers@biblibre.com> --- (In reply to Fridolin Somers from comment #45)
Enhancement not pushed to 23.05.x
Sure, I'll try backport. Any ES feature is welcome ;) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #48 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Fridolin Somers from comment #47)
(In reply to Fridolin Somers from comment #45)
Enhancement not pushed to 23.05.x
Sure, I'll try backport. Any ES feature is welcome ;)
Ping? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_23_05_candidate --- Comment #49 from Fridolin Somers <fridolin.somers@biblibre.com> --- I'll work on it -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=5534 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 --- Comment #50 from Nick Clemens <nick@bywatersolutions.com> --- Current patches apply cleanly to 23.05.x and work in my testing - can this be considered for backport? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Laura O'Neil <laura@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |laura@bywatersolutions.com --- Comment #51 from Laura O'Neil <laura@bywatersolutions.com> --- +1 for backport to 23.05, this would be very helpful to a lot of libraries. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=37137 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC|martin.renvoize@ptfs-europe | |.com | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Documentation| |Caroline Cyr La Rose contact| | Documentation| |https://gitlab.com/koha-com submission| |munity/koha-manual/-/merge_ | |requests/953 CC| |caroline.cyr-la-rose@inlibr | |o.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lisette@bywatersolutions.co | |m Keywords|release-notes-needed | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Rebecca Coert <rcoert@arlingtonva.us> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rcoert@arlingtonva.us -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=39598 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27153 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wainuiwitikapark@catalyst.n | |et.nz Resolution|--- |FIXED Status|Pushed to main |RESOLVED --- Comment #52 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- Not backporting to 22.11 23.05 is no longer supported -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org