[Bug 23204] New: Script for exporting Elasticsearch mappings to YAML
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Bug ID: 23204 Summary: Script for exporting Elasticsearch mappings to YAML Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: enhancement Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: alex.arnaud@biblibre.com Target Milestone: --- Sometimes we need to report librarian's mapping changes in DB to the mappings.yaml. This can be long and tedious. So, here is a script that do the job. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Alex Arnaud <alex.arnaud@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |alex.arnaud@biblibre.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #1 from Alex Arnaud <alex.arnaud@biblibre.com> --- Created attachment 90999 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=90999&action=edit Bug 23204 - script for exporting Elasticsearch mappings Test plan: - launch perl misc/search_tools/export_elasticsearch_mappings.pl > /path/to/my_mappings.yaml - set koha-conf.elasticsearch_index_mappings to /path/to/my_mappings.yaml, - go to admin -> Search engine configuration, - click on "Reset mappins", - check that your search fields and mappings are as expected -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Alex Arnaud <alex.arnaud@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Bouzid Fergani <bouzid.fergani@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #90999|0 |1 is obsolete| | --- Comment #2 from Bouzid Fergani <bouzid.fergani@inlibro.com> --- Created attachment 91350 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=91350&action=edit Bug 23204 - script for exporting Elasticsearch mappings Test plan: - launch perl misc/search_tools/export_elasticsearch_mappings.pl > /path/to/my_mappings.yaml - set koha-conf.elasticsearch_index_mappings to /path/to/my_mappings.yaml, - go to admin -> Search engine configuration, - click on "Reset mappins", - check that your search fields and mappings are as expected Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Bouzid Fergani <bouzid.fergani@inlibro.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bouzid.fergani@inlibro.com Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com Status|Signed Off |Failed QA --- Comment #3 from Nick Clemens <nick@bywatersolutions.com> --- This is great, few things: 1 - This should be a subroutine that can be unit tested (also then we can easily integrate into staff client and allow users to export mappings) 2 - This should take preserves weights specified in the client -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Alex Arnaud <alex.arnaud@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #91350|0 |1 is obsolete| | --- Comment #4 from Alex Arnaud <alex.arnaud@biblibre.com> --- Created attachment 92022 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=92022&action=edit Bug 23204 - script for exporting Elasticsearch mappings Test plan: - launch perl misc/search_tools/export_elasticsearch_mappings.pl > /path/to/my_mappings.yaml - set koha-conf.elasticsearch_index_mappings to /path/to/my_mappings.yaml, - go to admin -> Search engine configuration, - click on "Reset mappins", - check that your search fields and mappings are as expected Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #5 from Alex Arnaud <alex.arnaud@biblibre.com> --- Created attachment 92023 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=92023&action=edit Bug 23204: move code in a unit tested sub -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #6 from Alex Arnaud <alex.arnaud@biblibre.com> --- Created attachment 92024 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=92024&action=edit Bug 23204: export field weights -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Alex Arnaud <alex.arnaud@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #7 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- misc/search_tools/export_elasticsearch_mappings.pl is missing exec permissions : chmod a+x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #8 from Alex Arnaud <alex.arnaud@biblibre.com> --- Created attachment 92174 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=92174&action=edit Bug 23204: add exec permission on the script -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #9 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Really cool script. Just, the generated YAML does not ensure sorting order. Fields must be sorted by name and mappings by marc type (at least). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #10 from Alex Arnaud <alex.arnaud@biblibre.com> --- Created attachment 93524 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=93524&action=edit Bug 23204: sort search fields by name -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Séverine Queune <severine.queune@bulac.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |severine.queune@bulac.fr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 e.betemps <e.betemps@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |e.betemps@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #11 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Created attachment 95986 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=95986&action=edit Bug 23204: (follow-up) also sort by marc field Test plan : Launching script several times and see that marc field is alway ordered increasing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #12 from Nick Clemens <nick@bywatersolutions.com> --- Question: Should we only load/export the mappings for the marc_type of the instance? Including the normarc and unimarc for marc21 makes the file harder to read. Same the other way round to :-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mkstephens@lancasterseminar | |y.edu -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #13 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- (In reply to Nick Clemens from comment #12)
Question: Should we only load/export the mappings for the marc_type of the instance? Including the normarc and unimarc for marc21 makes the file harder to read. Same the other way round to :-)
There is the --type arg in the script to export only one marc flavour. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #92022|0 |1 is obsolete| | --- Comment #14 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 100960 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100960&action=edit Bug 23204: script for exporting Elasticsearch mappings Test plan: - launch perl misc/search_tools/export_elasticsearch_mappings.pl > /path/to/my_mappings.yaml - set koha-conf.elasticsearch_index_mappings to /path/to/my_mappings.yaml, - go to admin -> Search engine configuration, - click on "Reset mappins", - check that your search fields and mappings are as expected Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Fixed commit message -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #92023|0 |1 is obsolete| | --- Comment #15 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 100961 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100961&action=edit Bug 23204: move code in a unit tested sub Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Test pass -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #92024|0 |1 is obsolete| | --- Comment #16 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 100962 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100962&action=edit Bug 23204: export field weights Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #92174|0 |1 is obsolete| | --- Comment #17 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 100963 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100963&action=edit Bug 23204: add exec permission on the script Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #93524|0 |1 is obsolete| | --- Comment #18 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 100964 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100964&action=edit Bug 23204: sort search fields by name Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #95986|0 |1 is obsolete| | --- Comment #19 from Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> --- Created attachment 100965 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=100965&action=edit Bug 23204: (follow-up) also sort by marc field Test plan : Launching script several times and see that marc field is alway ordered increasing Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Same order between runs -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bgkriegel@gmail.com Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #20 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I think the exported file must export exactly the same file as the one we embed with Koha. % perl misc/search_tools/export_elasticsearch_mappings.pl > /tmp/mappings.yaml % diff admin/searchengine/elasticsearch/mappings.yaml /tmp/mappings.yaml Should return no change. Don't you think? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #21 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 101742 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=101742&action=edit Bug 23204: Fix ordering of fields in exported file -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #22 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Jonathan Druart from comment #20)
I think the exported file must export exactly the same file as the one we embed with Koha.
% perl misc/search_tools/export_elasticsearch_mappings.pl > /tmp/mappings.yaml % diff admin/searchengine/elasticsearch/mappings.yaml /tmp/mappings.yaml Should return no change.
Don't you think?
Yes and no - ordering fixed most, and not adding undefined weights or facet_order, but there are some remaining differences: 1 - Facet order not set in default file 2 - LC-card-number - comes out in lower case because we have it twice, once for authorities and once for biblios - these get combined when adding to DB because they all go in to 'search_field' which doesn't separate bibs/auths - I don't think the matters. There does seem to be convention of Auth fields being capitalized and bibs fields not, but I don't believe we rely on this or need to 3 - Default mapping file is not strictly alphabetized - date-of-publication moves on export Probably a good compromise is to generate the mapping file from this bug once it passes QA and use the exported file as the default from now on. Consider it a cleanup of the file -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #23 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- (In reply to Nick Clemens from comment #22)
Probably a good compromise is to generate the mapping file from this bug once it passes QA and use the exported file as the default from now on. Consider it a cleanup of the file
+1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #100960|0 |1 is obsolete| | --- Comment #24 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 101850 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=101850&action=edit Bug 23204: script for exporting Elasticsearch mappings Test plan: - launch perl misc/search_tools/export_elasticsearch_mappings.pl > /path/to/my_mappings.yaml - set koha-conf.elasticsearch_index_mappings to /path/to/my_mappings.yaml, - go to admin -> Search engine configuration, - click on "Reset mappins", - check that your search fields and mappings are as expected Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Fixed commit message Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #100961|0 |1 is obsolete| | --- Comment #25 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 101851 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=101851&action=edit Bug 23204: move code in a unit tested sub Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Test pass Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #100962|0 |1 is obsolete| | --- Comment #26 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 101852 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=101852&action=edit Bug 23204: export field weights Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #100963|0 |1 is obsolete| | --- Comment #27 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 101853 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=101853&action=edit Bug 23204: add exec permission on the script Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #100964|0 |1 is obsolete| | --- Comment #28 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 101854 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=101854&action=edit Bug 23204: sort search fields by name Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #100965|0 |1 is obsolete| | --- Comment #29 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 101855 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=101855&action=edit Bug 23204: (follow-up) also sort by marc field Test plan : Launching script several times and see that marc field is alway ordered increasing Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Same order between runs Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #101742|0 |1 is obsolete| | --- Comment #30 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 101856 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=101856&action=edit Bug 23204: Fix ordering of fields in exported file Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #31 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 101857 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=101857&action=edit Bug 23204: Update the mappings.yaml file with the output generated We need to be consistent and provide the mappings.yaml file generated by the script. Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #32 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 101858 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=101858&action=edit Bug 23204: Adjust copyright and license Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #33 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Jonathan Druart from comment #31)
Created attachment 101857 [details] [review] Bug 23204: Update the mappings.yaml file with the output generated
We need to be consistent and provide the mappings.yaml file generated by the script.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Alex, Nick, can you confirm you agree with that? If so I think we need a test to make sure further changes will be put into the correct place in the file. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |19482 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=19482 [Bug 19482] Elasticsearch - prevent removal / editing of required indexes -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #34 from Alex Arnaud <alex.arnaud@biblibre.com> ---
Alex, Nick, can you confirm you agree with that?
If so I think we need a test to make sure further changes will be put into the correct place in the file.
I agree ! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |24997 --- Comment #35 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Jonathan Druart from comment #33)
(In reply to Jonathan Druart from comment #31)
Created attachment 101857 [details] [review] [review] Bug 23204: Update the mappings.yaml file with the output generated
We need to be consistent and provide the mappings.yaml file generated by the script.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Alex, Nick, can you confirm you agree with that?
If so I think we need a test to make sure further changes will be put into the correct place in the file.
See bug 24997. Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24997 [Bug 24997] Test to ensure mappings.yaml is in the correct order -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |20.05.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #36 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Nice work everyone! Pushed to master for 20.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #37 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- We would really like this one in 19.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #38 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 102010 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=102010&action=edit Bug 23204: Fix tests swapping fields (they are not ordered) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Joy Nelson <joy@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |joy@bywatersolutions.com --- Comment #39 from Joy Nelson <joy@bywatersolutions.com> --- enhancement not backported to 19.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |frederic@tamil.fr --- Comment #40 from Frédéric Demians <frederic@tamil.fr> --- Note : Since bug 20589, two fields have been added to 'search_field' table: staff_client and opac. Those fields are lacking in mappings.yaml. Its means that if this script is used to export and import (Reset mapping in Search engine configuration) indexing rules, the searchability (staff/opac) of fields will be lost. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |25325 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25325 [Bug 25325] ElasticSearch mapping export lacks staff_client/opac fields -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |25337 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25337 [Bug 25337] Export/import of Elasticsearch configuration is lacking information -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #41 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- (In reply to Frédéric Demians from comment #40)
Note : Since bug 20589, two fields have been added to 'search_field' table: staff_client and opac. Those fields are lacking in mappings.yaml. Its means that if this script is used to export and import (Reset mapping in Search engine configuration) indexing rules, the searchability (staff/opac) of fields will be lost.
I've opened a new bug report: Bug 25337 - Export/import of Elasticsearch configuration is lacking information -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Script for exporting |Add script for exporting |Elasticsearch mappings to |elasticsearch mappings to |YAML |YAML CC| |martin.renvoize@ptfs-europe | |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #42 from Alex Arnaud <alex.arnaud@biblibre.com> --- (In reply to Jonathan Druart from comment #31)
Created attachment 101857 [details] [review] Bug 23204: Update the mappings.yaml file with the output generated
We need to be consistent and provide the mappings.yaml file generated by the script.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
I think i agreed a little to fast this commit. Removing quote on field number starting with 0 (030) make that number is badly interpreted (as octal). So this mapping: coden: label: coden mappings: - facet: '' marc_field: 030 marc_type: marc21 sort: ~ suggestible: '' makes coden mapped with field 24 Indexing fail: root@2dc1f5ace3f5:koha(master)$ perl misc/search_tools/rebuild_elasticsearch.pl -b -a -d -v [1324] Checking state of biblios index [1324] Dropping and recreating biblios index [1324] Checking state of authorities index [1324] Dropping and recreating authorities index [1324] Indexing biblios [1324] Committing final records... Invalid MARC field expression: 24 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #43 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Alex Arnaud from comment #42)
(In reply to Jonathan Druart from comment #31)
I think i agreed a little to fast this commit. Removing quote on field number starting with 0 (030) make that number is badly interpreted (as octal).
I experienced this once, however, I cannot recreate it now, what is your setup when this happens? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #44 from Frédéric Demians <frederic@tamil.fr> --- This issue may depend on the YAML parser used by default. I can't neither reproduce it on my setup. Replacing "use YAML" with "use YAML::Syck" could be a solution. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #45 from Alex Arnaud <alex.arnaud@biblibre.com> --- (In reply to Nick Clemens from comment #43)
(In reply to Alex Arnaud from comment #42)
I experienced this once, however, I cannot recreate it now, what is your setup when this happens?
Now i can reproduce it almost each time: - start a fresh koha container (koha-testing-docker): ku-es5 or reset_all in an existing one, - Change searchengine syspref to "Elasticsearch", - Go to Admin -> Search engine configuration (Elasticsearch), - mapping ok (coden => 030, code-institution => 040 etc...) - reset mapping - mapping Ko (coden => 24, code-institution => 32 etc...) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #46 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Could someone deal with that? At least to open a new bug report. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 anne-cecile.bories@iepg.fr changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |anne-cecile.bories@iepg.fr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #47 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I don't recreate with KOHA_IMAGE=master-bionic and docker-compose.es6.yml -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 --- Comment #48 from Frédéric Demians <frederic@tamil.fr> --- If there is a new bug, assign it to me. I will do my best to reproduce it and fix it. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Add script for exporting |Add script for exporting |elasticsearch mappings to |Elasticsearch mappings to |YAML |YAML -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |22639 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22639 [Bug 22639] Elasticsearch ability export mappings and facets settings -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |26832 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 [Bug 26832] Elasticsearch mappings export should use UTF-8 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41669 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |41670 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41670 [Bug 41670] Elasticsearch mappings do not match mappings export -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org