[Bug 19707] New: "merge", "revert" and "add" operations for synchronizing Elasticsearch mappings stored in database with mappings.yaml
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 Bug ID: 19707 Summary: "merge", "revert" and "add" operations for synchronizing Elasticsearch mappings stored in database with mappings.yaml Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: glasklas@gmail.com It would be nice to have the option to synchronize possible upstream mappings with local onces, without overwritning local changes, so wrote this patch to "merge", "revert" and "add" operations for synchronizing Elasticsearch mappings stored in database with mappings.yaml -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|"merge", "revert" and "add" |Add new operations for |operations for |synchronizing Elasticsearch |synchronizing Elasticsearch |mappings |mappings stored in database | |with mappings.yaml | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 --- Comment #1 from David Gustafsson <glasklas@gmail.com> --- Created attachment 69414 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=69414&action=edit Bug 19707 - Add new operations for synchronizing Elasticsearch mappings Add new "merge", "revert" and "add" operations for synchronizing Elasticsearch mappings stored in database with mappings.yaml -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 --- Comment #2 from David Gustafsson <glasklas@gmail.com> --- "add": Only new fields will be added (present in mappings.yaml but not in database), mappings for fields stored in database will be left untouched. "revert": Mappings for all fields in mappings.yaml will be reverted, mappings restored to default onces, and local modifications of those mappings removed. "merge": New mappings for fields present in mappings.yaml will be merged in, keeping any local modifications of marc targets (this could also include old problematic mappings later fixed in mappings.yaml, so should be used with some caution). -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |severine.queune@bulac.fr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 --- Comment #3 from David Gustafsson <glasklas@gmail.com> --- Just saw that commit message contained no How to test, I should probably fix this and will try to do so in the near future. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 --- Comment #4 from Séverine Queune <severine.queune@bulac.fr> --- Hi David, I take a look to your patch to see if I can manage without a test plan but I failed :) I can test it in the next few days if you can provide a test plan ! -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |glasklas@gmail.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #69414|0 |1 is obsolete| | --- Comment #5 from David Gustafsson <glasklas@gmail.com> --- Created attachment 74819 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=74819&action=edit Bug 19707 - Add new operations for synchronizing Elasticsearch mappings Add new "merge", "revert" and "add" operations for synchronizing Elasticsearch mappings stored in database with mappings.yaml How to test: 1) Make sure mappings.yaml contains mappings not present in database. (Also be aware that steps below will result in changed search engine configuration, so use a test environment and/or make sure to backup database before starting) 2) Go to "Administration" and "Search Engine Configuration" 3) Append ?i_know_what_i_am_doing=1&op=add to url and perform request in browser 4) Vefify that any new mappings in mappings.yaml has been added, but no current mappings has been modified 5) Edit mappings.yaml and add a new mapping for some field (new item below "mappings:" key) 6) Add a new mapping for the same field in the database (through admin ui) 7) Append ?i_know_what_i_am_doing=1&op=merge to url and perform request in browser 8) Verify that the new mapping from mappings.yaml has been added for this field, while the field added in database was preserved 10) Append ?i_know_what_i_am_doing=1&op=revert to url and perform request in browser 11) Vefify that the mapping previously added the field in database has been deleted, and the mappings for this field are identical to those in mappings.yaml Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #74819|0 |1 is obsolete| | --- Comment #6 from David Gustafsson <glasklas@gmail.com> --- Created attachment 74820 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=74820&action=edit Bug 19707 - Add new operations for synchronizing Elasticsearch mappings Add new "merge", "revert" and "add" operations for synchronizing Elasticsearch mappings stored in database with mappings.yaml How to test: 1) Make sure mappings.yaml contains mappings not present in database. (Also be aware that steps below will result in changed search engine configuration, so use a test environment and/or make sure to backup database before starting) 2) Go to "Administration" and "Search Engine Configuration" 3) Append ?i_know_what_i_am_doing=1&op=add to url and perform request in browser 4) Verify that any new mappings in mappings.yaml has been added, but no current mappings has been modified or deleted 5) Edit mappings.yaml and add a new mapping for some field (new item below "mappings:" key) 6) Add a new mapping for the same field in the database (through admin ui) 7) Append ?i_know_what_i_am_doing=1&op=merge to url and perform request in browser 8) Verify that the new mapping from mappings.yaml has been added for this field, while the field added in database was preserved 10) Append ?i_know_what_i_am_doing=1&op=revert to url and perform request in browser 11) Verify that the mapping previously added the field in database has been deleted, and the mappings for this field are identical to those in mappings.yaml Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 --- Comment #7 from David Gustafsson <glasklas@gmail.com> --- Corrected some spelling mistakes in commit message. @Séverine Queune: thankgs for taking an interest, there now is a test-plan. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Patch doesn't apply --- Comment #8 from Séverine Queune <severine.queune@bulac.fr> --- Hi David, Thank you for your dev, it's a useful one so it's natural to be interested by it :) I can't apply it on master branch (17.12.00.040) : 74820 - Bug 19707 - Add new operations for synchronizing Elasticsearch mappings Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 19707 - Add new operations for synchronizing Elasticsearch mappings Using index info to reconstruct a base tree... M Koha/SearchEngine/Elasticsearch.pm Falling back to patching base and 3-way merge... Auto-merging Koha/SearchEngine/Elasticsearch.pm CONFLICT (content): Merge conflict in Koha/SearchEngine/Elasticsearch.pm Failed to merge in the changes. Patch failed at 0001 Bug 19707 - Add new operations for synchronizing Elasticsearch mappings The copy of the patch that failed is found in: /var/repositories/koha/.git/rebase-apply/patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #74820|0 |1 is obsolete| | --- Comment #9 from David Gustafsson <glasklas@gmail.com> --- Created attachment 75107 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=75107&action=edit Bug 19707 - Add new operations for synchronizing Elasticsearch mappings Add new "merge", "revert" and "add" operations for synchronizing Elasticsearch mappings stored in database with mappings.yaml -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #75107|0 |1 is obsolete| | --- Comment #10 from David Gustafsson <glasklas@gmail.com> --- Created attachment 75108 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=75108&action=edit Bug 19707 - Add new operations for synchronizing Elasticsearch mappings Add new "merge", "revert" and "add" operations for synchronizing Elasticsearch mappings stored in database with mappings.yaml How to test: 1) Make sure mappings.yaml contains mappings not present in database. (Also be aware that steps below will result in changed search engine configuration, so use a test environment and/or make sure to backup database before starting) 2) Go to "Administration" and "Search Engine Configuration" 3) Append ?i_know_what_i_am_doing=1&op=add to url and perform request in browser 4) Verify that any new mappings in mappings.yaml has been added, but no current mappings has been modified or deleted 5) Edit mappings.yaml and add a new mapping for some field (new item below "mappings:" key) 6) Add a new mapping for the same field in the database (through admin ui) 7) Append ?i_know_what_i_am_doing=1&op=merge to url and perform request in browser 8) Verify that the new mapping from mappings.yaml has been added for this field, while the field added in database was preserved 10) Append ?i_know_what_i_am_doing=1&op=revert to url and perform request in browser 11) Verify that the mapping previously added the field in database has been deleted, and the mappings for this field are identical to those in mappings.yaml Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 --- Comment #11 from David Gustafsson <glasklas@gmail.com> --- Rebased against Koha master. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #75108|0 |1 is obsolete| | --- Comment #12 from David Gustafsson <glasklas@gmail.com> --- Created attachment 79109 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=79109&action=edit Bug 19707 - Add new operations for synchronizing Elasticsearch mappings Add new "merge", "revert" and "add" operations for synchronizing Elasticsearch mappings stored in database with mappings.yaml How to test: 1) Make sure mappings.yaml contains mappings not present in database. (Also be aware that steps below will result in changed search engine configuration, so use a test environment and/or make sure to backup database before starting) 2) Go to "Administration" and "Search Engine Configuration" 3) Append ?i_know_what_i_am_doing=1&op=add to url and perform request in browser 4) Verify that any new mappings in mappings.yaml has been added, but no current mappings has been modified or deleted 5) Edit mappings.yaml and add a new mapping for some field (new item below "mappings:" key) 6) Add a new mapping for the same field in the database (through admin ui) 7) Append ?i_know_what_i_am_doing=1&op=merge to url and perform request in browser 8) Verify that the new mapping from mappings.yaml has been added for this field, while the field added in database was preserved 10) Append ?i_know_what_i_am_doing=1&op=revert to url and perform request in browser 11) Verify that the mapping previously added the field in database has been deleted, and the mappings for this field are identical to those in mappings.yaml Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 --- Comment #13 from David Gustafsson <glasklas@gmail.com> --- Something when wrong with the previous rebase, so rebased again and added buttons for all the new operations. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #79109|0 |1 is obsolete| | --- Comment #14 from David Gustafsson <glasklas@gmail.com> --- Created attachment 79112 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=79112&action=edit Bug 19707 - Add new operations for synchronizing Elasticsearch mappings Add new "merge", "revert" and "add" operations for synchronizing Elasticsearch mappings stored in database with mappings.yaml How to test: 1) Make sure mappings.yaml contains mappings not present in database. (Also be aware that steps below will result in changed search engine configuration, so use a test environment and/or make sure to backup database before starting) 2) Go to "Administration" and "Search Engine Configuration" 3) Click "Add mappings" and confirm 4) Verify that any new mappings in mappings.yaml has been added, but no current mappings has been modified or deleted 5) Edit mappings.yaml and add a new mapping for some field (new item below "mappings:" key) 6) Add a new mapping for the same field in the database (through admin ui) 7) Click "Merge mappings" and confirm 8) Verify that the new mapping from mappings.yaml has been added for this field, while the field added in database was preserved 10) Click "Revert mappings" and confirm 11) Verify that the mapping previously added the field in database has been deleted, and the mappings for this field are identical to those in mappings.yaml Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 --- Comment #15 from David Gustafsson <glasklas@gmail.com> --- Fixed commit-message to reflect latest changes. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #79112|0 |1 is obsolete| | --- Comment #16 from David Gustafsson <glasklas@gmail.com> --- Created attachment 82795 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82795&action=edit Bug 19707 - Add new operations for synchronizing Elasticsearch mappings Add new "merge", "revert" and "add" operations for synchronizing Elasticsearch mappings stored in database with mappings.yaml How to test: 1) Make sure mappings.yaml contains mappings not present in database. (Also be aware that steps below will result in changed search engine configuration, so use a test environment and/or make sure to backup database before starting) 2) Go to "Administration" and "Search Engine Configuration" 3) Click "Add mappings" and confirm 4) Verify that any new mappings in mappings.yaml has been added, but no current mappings has been modified or deleted 5) Edit mappings.yaml and add a new mapping for some field (new item below "mappings:" key) 6) Add a new mapping for the same field in the database (through admin ui) 7) Click "Merge mappings" and confirm 8) Verify that the new mapping from mappings.yaml has been added for this field, while the field added in database was preserved 10) Click "Revert mappings" and confirm 11) Verify that the mapping previously added the field in database has been deleted, and the mappings for this field are identical to those in mappings.yaml Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 --- Comment #17 from David Gustafsson <glasklas@gmail.com> --- Rebase against master. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 Joonas Kylmälä <joonas.kylmala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joonas.kylmala@helsinki.fi Status|Needs Signoff |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #82795|0 |1 is obsolete| | --- Comment #18 from David Gustafsson <glasklas@gmail.com> --- Created attachment 110054 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=110054&action=edit Bug 19707 - Add new operations for synchronizing Elasticsearch mappings Add new "merge", "revert" and "add" operations for synchronizing Elasticsearch mappings stored in database with mappings.yaml How to test: 1) Make sure mappings.yaml contains mappings not present in database. (Also be aware that steps below will result in changed search engine configuration, so use a test environment and/or make sure to backup database before starting) 2) Go to "Administration" and "Search Engine Configuration" 3) Click "Add mappings" and confirm 4) Verify that any new mappings in mappings.yaml has been added, but no current mappings has been modified or deleted 5) Edit mappings.yaml and add a new mapping for some field (new item below "mappings:" key) 6) Add a new mapping for the same field in the database (through admin ui) 7) Click "Merge mappings" and confirm 8) Verify that the new mapping from mappings.yaml has been added for this field, while the field added in database was preserved 10) Click "Revert mappings" and confirm 11) Verify that the mapping previously added the field in database has been deleted, and the mappings for this field are identical to those in mappings.yaml Sponsored-by: Gothenburg University Library -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 David Gustafsson <glasklas@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Needs Signoff --- Comment #19 from David Gustafsson <glasklas@gmail.com> --- Rebased against master. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #20 from Séverine Queune <severine.queune@bulac.fr> --- I've got an issue with merging option (steps 6-8) : I can add a new mapping in UI, no problem for saving it but after merging, the field disappear form UI and (obviously) it's not added to yaml file. I didn't succeed to revert the mapping too, but maybe this option is only available combined with the merge option ? Just for my curiosity, how the modifications are stored to enable a revert ? This is such a great feature !!! Can't wait to see it in master ! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19707 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ere.maijala@helsinki.fi -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon@bugs.koha-community.org