[Bug 30836] New: Elasticsearch mappings.yaml is out of sync with upgraded databases
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 Bug ID: 30836 Summary: Elasticsearch mappings.yaml is out of sync with upgraded databases Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: martin.renvoize@ptfs-europe.com QA Contact: testopia@bugs.koha-community.org We discovered on IRC today that the mappings.yaml default mappings for ES indexing has fallen out of sync with upgraded databases.. We need to create a DB update for each mappings change and we haven't been doing so. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart+koha@gmail. | |com, | |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 --- Comment #1 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- This also needs a guideline for QA people to follow so we don't fall out of sync again in the future. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 --- Comment #2 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- http://irc.koha-community.org/koha/2022-05-24#i_2426285 for irc reference -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 --- Comment #3 from Nick Clemens <nick@bywatersolutions.com> --- Out of the box it does seem to work fine adding a mapping to the db directly, however, it relies on dynamic mapping being enabled: https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-mapp... Otherwise, we will need to update the indexes in ES as well -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ere.maijala@helsinki.fi, | |nugged@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 --- Comment #4 from Tomás Cohen Arazi <tomascohen@gmail.com> --- We ceratinly need to define a good way to patch things in production for the regular user, leaving room so we don't break more advanced users setups. If this is not doable, then we need to document better how to reset the indexes, or provide a tool to let them know they are out of sync. This is a blocker for several enhancements. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 --- Comment #5 from Ere Maijala <ere.maijala@helsinki.fi> --- I'm not sure if I can come up with a good solution. Automatically updating current mappings can create problems when the defaults have been customized. What we might be able to do is to check, field by field, if the mappings have been customized, and apply any changes only on top of old defaults. I'm not sure if that's feasible in real life, but worth a thought. At least it should be possible to check that any mappings required by Koha are available regardless of their configuration, and add any that are missing completely. Perhaps a warning about updated mappings during upgrade would be useful as well. I think that in many cases requiring resetting to defaults would be too destructive. Having to redo all customizations could be pretty a lot of work unless you manage them in a customized mappings.yaml or something. It would be useful to have a tool that would show the differences between current and default mappings. This would make it easier to determine any manual changes needed or useful. All in all, I'm not sure I like having the database part of the configuration. Sure, the UI makes it easier to start changing the mappings, but if you need to do more, it's cumbersome. Also changing the settings can have consequences that you need to be aware of, and sometimes I feel it should not be that easy to change them. Regardless of how the mappings update is managed, any changes will need to trigger a prominent notification that reindexing is required. Sorry for this being a collection of random thoughts... -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- (In reply to Ere Maijala from comment #5)
I'm not sure if I can come up with a good solution. Automatically updating current mappings can create problems when the defaults have been customized.
This reminds me that we have this problem in other parts of Koha as well, and we've never found a good solution. For example, MARC Bibliographic Frameworks. Also, "Koha to MARC mapping". (In reply to Ere Maijala from comment #5)
What we might be able to do is to check, field by field, if the mappings have been customized, and apply any changes only on top of old defaults. I'm not sure if that's feasible in real life, but worth a thought. At least it should be possible to check that any mappings required by Koha are available regardless of their configuration, and add any that are missing completely. Perhaps a warning about updated mappings during upgrade would be useful as well.
Actually... I think that this is probably the way to go. I'm not familiar with the Elasticsearch mappings setup, but in theory couldn't you compare default values against user-generated values? And then update everything but those? And then maybe have some way of telling users which values are default and which values aren't in a visual way? I imagine that might be easier for Elasticsearch mappings than it would for say "MARC Bibliographic Frameworks". -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 Stefan Berndtsson <stefan.berndtsson@ub.gu.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |stefan.berndtsson@ub.gu.se --- Comment #7 from Stefan Berndtsson <stefan.berndtsson@ub.gu.se> --- Isn't this what Bug 19707 is supposed to handle? I know it has failed QA for now though, so some fixing would be necessary, but if I remember correctly it adds sync/reset options for mappings. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 --- Comment #8 from Ere Maijala <ere.maijala@helsinki.fi> --- Looks like Bug 19707 would indeed provide a lot of the functionality. What remains would be automatic updating of non-customized fields and prominent message during upgrade that prompts to merge any changes manually. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30836 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org