[Bug 26832] New: Elasticsearch mappings export should use UTF-8
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 Bug ID: 26832 Summary: Elasticsearch mappings export should use UTF-8 Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: fridolin.somers@biblibre.com Script misc/search_tools/export_elasticsearch_mappings.pl allows to export current search engine configuration into a YAML file. This export should use UTF-8 encoding, like other exports. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |fridolin.somers@biblibre.co |ity.org |m Depends on| |23204 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 [Bug 23204] Add script for exporting Elasticsearch mappings to YAML -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 Fridolin SOMERS <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 --- Comment #1 from Fridolin SOMERS <fridolin.somers@biblibre.com> --- Created attachment 112546 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112546&action=edit Bug 26832: Make Elasticsearch mappings export use UTF-8 Script misc/search_tools/export_elasticsearch_mappings.pl allows to export current search engine configuration into a YAML file. This export should use UTF-8 encoding, like other exports. Test plan : 1) Go to Administration > Search engine configuration (Elasticsearch) 2) Edit a field label to use a diacrtic, for example local-number => Numéro 3) Save 4) Edit file etc/koha-conf.xml to enable 'elasticsearch_index_mappings' 5) Export mappings to file via misc/search_tools/export_elasticsearch_mappings.pl -t $MARCFLAVOUR 6) Reset memcached and plack 7) Back to Administration > Search engine configuration (Elasticsearch) 8) Click on 'Reset Mappings' and accept 9) Look at field 'local-number' => Without patch diacritic 'é' is broken 10) You may try with an emoji B-) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #112546|0 |1 is obsolete| | --- Comment #2 from David Nind <david@davidnind.com> --- Created attachment 112813 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112813&action=edit Bug 26832: Make Elasticsearch mappings export use UTF-8 Script misc/search_tools/export_elasticsearch_mappings.pl allows to export current search engine configuration into a YAML file. This export should use UTF-8 encoding, like other exports. Test plan : 1) Go to Administration > Search engine configuration (Elasticsearch) 2) Edit a field label to use a diacrtic, for example local-number => Numéro 3) Save 4) Edit file etc/koha-conf.xml to enable 'elasticsearch_index_mappings' 5) Export mappings to file via misc/search_tools/export_elasticsearch_mappings.pl -t $MARCFLAVOUR 6) Reset memcached and plack 7) Back to Administration > Search engine configuration (Elasticsearch) 8) Click on 'Reset Mappings' and accept 9) Look at field 'local-number' => Without patch diacritic 'é' is broken 10) You may try with an emoji B-) Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #3 from David Nind <david@davidnind.com> --- I hope I tested this right! I followed the test plan without applying the patch. I got an error with resetting the mappings at steps 8-9: DBIx::Class::Storage::DBI::_dbh_execute(): Incorrect string value: '\xE9ro' for column 'label' at row 1 at /kohadevbox/koha/Koha/Objects.pm line 114 at /usr/share/perl5/DBIx/Class/Exception.pm line 77 Notes for testing using koha-testing-docker: 1. Before starting testing setup elastic search as the search engine: i. Enable Elasticsearch: system preference SearchEngine ii. Reindex: koha-elasticsearch --rebuild -b -a kohadev 2. For step 4: i. Uncomment in /etc/koha/sites/kohadev/koha-conf.xml: <elasticsearch_index_mappings> ii. Note the location of the file: /etc/koha/sites/kohadev/searchengine/elasticsearch/mappings.yaml iii. Create the directories: searchengine and elasticsearch 3. For step 5: i. misc/search_tools/export_elasticsearch_mappings.pl -t marc21 > /etc/koha/sites/kohadev/searchengine/elasticsearch/mappings.yaml ii. Check that the mappings.yaml file was created iii. Change ownership: chown -R kohadev-koha:kohadev-koha /etc/koha/sites/kohadev/searchengine 4. I got errors in the user interface, a trace error page when trying to Reset Mappings, and when trying to reindex. 5. I got past this by: i. applying the patch ii. disabling <elasticsearch_index_mappings> in koha-conf.xml iii. flush_memcached, restart_all iv. Reset Mappings v. rerunning through the test plan (see notes 2 and 3 above) 6. To check if the file is UTF-8 use uchardet command: i. Install uchardet: apt-get install uchardet ii. Check the file format: uchardet /etc/koha/sites/kohadev/searchengine/elasticsearch/mappings.yaml iii. It should show as being UTF-8 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #112813|0 |1 is obsolete| | --- Comment #4 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 112899 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112899&action=edit Bug 26832: Make Elasticsearch mappings export use UTF-8 Script misc/search_tools/export_elasticsearch_mappings.pl allows to export current search engine configuration into a YAML file. This export should use UTF-8 encoding, like other exports. Test plan : 1) Go to Administration > Search engine configuration (Elasticsearch) 2) Edit a field label to use a diacrtic, for example local-number => Numéro 3) Save 4) Edit file etc/koha-conf.xml to enable 'elasticsearch_index_mappings' 5) Export mappings to file via misc/search_tools/export_elasticsearch_mappings.pl -t $MARCFLAVOUR 6) Reset memcached and plack 7) Back to Administration > Search engine configuration (Elasticsearch) 8) Click on 'Reset Mappings' and accept 9) Look at field 'local-number' => Without patch diacritic 'é' is broken 10) You may try with an emoji B-) Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Status|Signed Off |Passed QA --- Comment #5 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |20.11.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 --- Comment #6 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Pushed to master for 20.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lucas@bywatersolutions.com --- Comment #7 from Lucas Gass <lucas@bywatersolutions.com> --- backported to 20.05.x for 20.05.06 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aleisha@catalyst.net.nz --- Comment #8 from Aleisha Amohia <aleisha@catalyst.net.nz> --- missing dependencies, not backported to 19.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED Version(s)|20.11.00 |20.11.00, 20.05.06 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 Bug 26832 depends on bug 23204, which changed state. Bug 23204 Summary: Add script for exporting Elasticsearch mappings to YAML https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23204 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26832 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=32012 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org