[Bug 17255] New: Upgrade Elastic Search code to work with version 2.4+
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Bug ID: 17255 Summary: Upgrade Elastic Search code to work with version 2.4+ Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching Assignee: gmcharlt@gmail.com Reporter: chris@bigballofwax.co.nz QA Contact: testopia@bugs.koha-community.org Before building more things on Elastic (I have 3 things ready to go) We need to upgrade to work with the latest stable (2.4) This will involve getting a 2.4 cluster running, pointing Koha at it, and fixing any errors that occur. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Chris Cormack <chris@bigballofwax.co.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |12478 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 [Bug 12478] Elasticsearch support for Koha -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Mirko Tietgen <mirko@abunchofthings.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mirko@abunchofthings.net --- Comment #1 from Mirko Tietgen <mirko@abunchofthings.net> --- It will also need newer Catmandu than 0.9x I guess? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #2 from Chris Cormack <chris@bigballofwax.co.nz> --- That's what we will find out. I suspect so. But apparently 2.4 is more backward compatible than earlier 2.x versions. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #3 from Mirko Tietgen <mirko@abunchofthings.net> --- Created attachment 55205 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55205&action=edit libsearch-elasticsearch-perl_2.03-1_all.deb -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #4 from Mirko Tietgen <mirko@abunchofthings.net> --- Created attachment 55206 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=55206&action=edit libcatmandu-store-elasticsearch-perl_0.0501-1_all.deb -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #5 from Mirko Tietgen <mirko@abunchofthings.net> --- I have attached newer versions of the ES dependencies that I plan to submit to Debian. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #6 from Chris Cormack <chris@bigballofwax.co.nz> --- Awesome, thank you Mirko -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #7 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Chris Cormack from comment #0)
Before building more things on Elastic (I have 3 things ready to go)
Where are they? I have not found anything on the catalyst git repo. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #8 from Chris Cormack <chris@bigballofwax.co.nz> --- Three things ready to build that is. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Srdjan Jankovic <srdjan@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|gmcharlt@gmail.com |srdjan@catalyst.net.nz CC| |srdjan@catalyst.net.nz -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |17372 --- Comment #9 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Please build it on top of bug 17372. Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17372 [Bug 17372] Elasticsearch paths need to be standardized -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Josef Moravec <josef.moravec@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |josef.moravec@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Mason James <mtj@kohaaloha.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mtj@kohaaloha.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #10 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- It seems that the code is available at http://git.catalyst.co.nz/gw?p=koha.git;a=shortlog;h=refs/heads/wr262182 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #11 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Jonathan Druart from comment #10)
It seems that the code is available at http://git.catalyst.co.nz/gw?p=koha.git;a=shortlog;h=refs/heads/wr262182
What's the status of this branch? Is it considered as testable/wip/done? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #12 from Chris Cormack <chris@bigballofwax.co.nz> --- (In reply to Jonathan Druart from comment #11)
(In reply to Jonathan Druart from comment #10)
It seems that the code is available at http://git.catalyst.co.nz/gw?p=koha.git;a=shortlog;h=refs/heads/wr262182
What's the status of this branch? Is it considered as testable/wip/done?
It is testable, however * Facets most probably do not work (to be done) * We had to change Catmandu::Store::ElasticSearch::Bag https://github.com/LibreCat/Catmandu-Store-Elasticsearch/issues/10 But we would love some testing. (It is built on top of your bug to move the elasticsearch code (that is already in that branch)) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #13 from Chris Cormack <chris@bigballofwax.co.nz> --- Once the facets are working we will submit them as patches for sign off, but testing now would help catch any issues -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #14 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Note: The errors don't go to the error file but to stderr -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #15 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- perl misc/search_tools/rebuild_elastic_search.pl -d -a -b -v problem with :16 : :1: parser error : Start tag expected, '<' not found podigpodfigo ^ podigpodfigo at C4/Biblio.pm line 1353. No record passed at Koha/MetadataIterator.pm line 71. Something went wrong reading record for biblio Koha::Schema::Result::Biblio=HASH(0xd904770)->biblionumber: Can't bless non-reference value at Koha/BiblioUtils.pm line 70. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #16 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- It seems that I get facets for MARC21 but UNIMARC (?) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #17 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I did not install the second .deb, and it fails for me: % sudo dpkg -i attachment.cgi?id=55206 # libcatmandu-store-elasticsearch-perl_0.0501-1_all.deb (Reading database ... 83863 files and directories currently installed.) Preparing to unpack attachment.cgi?id=55206 ... Unpacking libcatmandu-store-elasticsearch-perl (0.0501-1) over (0.0304-1) ... dpkg: dependency problems prevent configuration of libcatmandu-store-elasticsearch-perl: libcatmandu-store-elasticsearch-perl depends on libcatmandu-perl (>= 1.01); however: Version of libcatmandu-perl on system is 0.9206-1. dpkg: error processing package libcatmandu-store-elasticsearch-perl (--install): dependency problems - leaving unconfigured Processing triggers for man-db (2.7.5-1) ... Errors were encountered while processing: libcatmandu-store-elasticsearch-perl % apt policy libcatmandu-perl libcatmandu-perl: Installed: 0.9206-1 Candidate: 0.9505-1 Version table: 0.9505-1 500 500 http://ftp.fr.debian.org/debian testing/main i386 Packages *** 0.9206-1 100 100 /var/lib/dpkg/status I am willing to test but I will need a test plan to upgrade my packages please. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #18 from Mirko Tietgen <mirko@abunchofthings.net> --- There is no newer version in the Debian repos yet. You can use https://apt.abunchofthings.net/debian/pool/main/libc/libcatmandu-perl/libcat... for the time being (and please report back if that works) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #19 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- % sudo dpkg -i libcatmandu-perl_1.0002-1_all.deb libsearch-elasticsearch-perl_2.03-1_all.deb libcatmandu-store-elasticsearch-perl_0.0501-1_all.deb (Reading database ... 83849 files and directories currently installed.) Preparing to unpack libcatmandu-perl_1.0002-1_all.deb ... Unpacking libcatmandu-perl (1.0002-1) over (0.9505-1) ... Preparing to unpack libsearch-elasticsearch-perl_2.03-1_all.deb ... Unpacking libsearch-elasticsearch-perl (2.03-1) over (2.03-1) ... Preparing to unpack libcatmandu-store-elasticsearch-perl_0.0501-1_all.deb ... Unpacking libcatmandu-store-elasticsearch-perl (0.0501-1) over (0.0304-1) ... dpkg: dependency problems prevent configuration of libcatmandu-perl: libcatmandu-perl depends on libmoox-role-logger-perl (>= 0.005); however: Package libmoox-role-logger-perl is not installed. libcatmandu-perl depends on libdata-uuid-perl (>= 1.217); however: Package libdata-uuid-perl is not installed. Version of libdata-uuid-perl on system, provided by libossp-uuid-perl:i386, is <none>. libcatmandu-perl depends on liburi-template-perl (>= 0.22); however: Version of liburi-template-perl on system is 0.18-1. dpkg: error processing package libcatmandu-perl (--install): dependency problems - leaving unconfigured Setting up libsearch-elasticsearch-perl (2.03-1) ... dpkg: dependency problems prevent configuration of libcatmandu-store-elasticsearch-perl: libcatmandu-store-elasticsearch-perl depends on libcatmandu-perl (>= 1.01); however: Package libcatmandu-perl is not configured yet. dpkg: error processing package libcatmandu-store-elasticsearch-perl (--install): dependency problems - leaving unconfigured Processing triggers for man-db (2.7.5-1) ... Errors were encountered while processing: libcatmandu-perl libcatmandu-store-elasticsearch-perl % sudo apt-get install -f The following additional packages will be installed: libalgorithm-combinatorics-perl libalgorithm-diff-xs-perl libanyevent-perl libapt-pkg-perl libasync-interrupt-perl libb-compiling-perl libb-hooks-op-check-entersubforcv-perl libb-hooks-op-check-perl libb-utils-perl libbareword-filehandles-perl libbit-vector-perl libcache-fastmmap-perl libcache-memcached-fast-perl libcairo-perl libclass-c3-xs-perl libclass-load-xs-perl libclass-xsaccessor-perl libclone-perl libcommon-sense-perl libcpanel-json-xs-perl libcrypt-eksblowfish-perl libcrypt-gcrypt-perl libcrypt-ssleay-perl libcss-minifier-xs-perl libdata-messagepack-perl libdata-util-perl libdate-calc-xs-perl libdatetime-format-mail-perl libdatetime-locale-perl libdatetime-perl libdatetime-timezone-perl libdbd-mysql-perl libdbd-sqlite2-perl libdbd-sqlite3-perl libdbi-perl libdevel-caller-perl libdevel-cover-perl libdevel-declare-perl libdevel-lexalias-perl libdevel-nytprof-perl libdevel-size-perl libdigest-sha-perl libev-perl libfcgi-perl libfile-fcntllock-perl libfilter-perl libfont-freetype-perl libgd-perl libglib-perl libgraphics-magick-perl libgssapi-perl libguard-perl libhtml-parser-perl libhtml-template-pro-perl libhtml-tidy-perl libhttp-tiny-perl libindirect-perl libio-pty-perl libjavascript-minifier-xs-perl libjson-xs-perl liblexical-sealrequirehints-perl liblingua-stem-snowball-perl liblinux-inotify2-perl liblist-moreutils-perl liblocale-gettext-perl libmarpa-r2-perl libmodule-metadata-perl libmoose-perl libmoox-role-logger-perl libmultidimensional-perl libnet-ssleay-perl libnet-z3950-zoom-perl libossp-uuid-perl libpackage-stash-xs-perl libpadwalker-perl libpango-perl libparams-classify-perl libparams-util-perl libparams-validate-perl libperl5.24 libperlio-gzip-perl libpng16-16 librdf-perl libreadonly-xs-perl libscalar-list-utils-perl libsereal-decoder-perl libsereal-encoder-perl libsocket6-perl libstring-crc32-perl libsub-identify-perl libsub-name-perl libtemplate-perl libterm-readkey-perl libtext-bidi-perl libtext-charwidth-perl libtext-csv-xs-perl libtext-iconv-perl libtext-soundex-perl libtext-unaccent-perl libtidy5 libtype-tiny-xs-perl libunicode-map-perl libunicode-utf8-perl libuuid-perl libvariable-magic-perl libversion-perl libwebp6 libxml-libxml-perl libxml-libxslt-perl libxml-parser-perl libyaml-libyaml-perl libyaml-syck-perl lintian perl perl-base perl-doc perl-modules-5.24 Suggested packages: libevent-perl libio-async-perl libmldbm-perl libbrowser-open-perl libtest-differences-perl graphicsmagick-dbg iwatch libscalar-number-perl libtemplate-perl-doc libtemplate-plugin-gd-perl libtemplate-plugin-xml-perl binutils-multiarch libterm-readline-gnu-perl | libterm-readline-perl-perl The following packages will be REMOVED: libcatmandu-marc-perl libcatmandu-perl libcatmandu-store-elasticsearch-perl libdata-alias-perl libdevel-beginlift-perl libparse-cpan-meta-perl The following NEW packages will be installed: libmoox-role-logger-perl libperl5.24 libpng16-16 libscalar-list-utils-perl libtidy5 libwebp6 perl-modules-5.24 The following packages will be upgraded: libalgorithm-combinatorics-perl libalgorithm-diff-xs-perl libanyevent-perl libapt-pkg-perl libasync-interrupt-perl libb-compiling-perl libb-hooks-op-check-entersubforcv-perl libb-hooks-op-check-perl libb-utils-perl libbareword-filehandles-perl libbit-vector-perl libcache-fastmmap-perl libcache-memcached-fast-perl libcairo-perl libclass-c3-xs-perl libclass-load-xs-perl libclass-xsaccessor-perl libclone-perl libcommon-sense-perl libcpanel-json-xs-perl libcrypt-eksblowfish-perl libcrypt-gcrypt-perl libcrypt-ssleay-perl libcss-minifier-xs-perl libdata-messagepack-perl libdata-util-perl libdate-calc-xs-perl libdatetime-format-mail-perl libdatetime-locale-perl libdatetime-perl libdatetime-timezone-perl libdbd-mysql-perl libdbd-sqlite2-perl libdbd-sqlite3-perl libdbi-perl libdevel-caller-perl libdevel-cover-perl libdevel-declare-perl libdevel-lexalias-perl libdevel-nytprof-perl libdevel-size-perl libdigest-sha-perl libev-perl libfcgi-perl libfile-fcntllock-perl libfilter-perl libfont-freetype-perl libgd-perl libglib-perl libgraphics-magick-perl libgssapi-perl libguard-perl libhtml-parser-perl libhtml-template-pro-perl libhtml-tidy-perl libhttp-tiny-perl libindirect-perl libio-pty-perl libjavascript-minifier-xs-perl libjson-xs-perl liblexical-sealrequirehints-perl liblingua-stem-snowball-perl liblinux-inotify2-perl liblist-moreutils-perl liblocale-gettext-perl libmarpa-r2-perl libmodule-metadata-perl libmoose-perl libmultidimensional-perl libnet-ssleay-perl libnet-z3950-zoom-perl libossp-uuid-perl libpackage-stash-xs-perl libpadwalker-perl libpango-perl libparams-classify-perl libparams-util-perl libparams-validate-perl libperlio-gzip-perl librdf-perl libreadonly-xs-perl libsereal-decoder-perl libsereal-encoder-perl libsocket6-perl libstring-crc32-perl libsub-identify-perl libsub-name-perl libtemplate-perl libterm-readkey-perl libtext-bidi-perl libtext-charwidth-perl libtext-csv-xs-perl libtext-iconv-perl libtext-soundex-perl libtext-unaccent-perl libtype-tiny-xs-perl libunicode-map-perl libunicode-utf8-perl libuuid-perl libvariable-magic-perl libversion-perl libxml-libxml-perl libxml-libxslt-perl libxml-parser-perl libyaml-libyaml-perl libyaml-syck-perl lintian perl perl-base perl-doc % pmvers Catmandu IO.c: loadable library and perl binaries are mismatched (got handshake key 0x8180080, needed 0x81c0080) % sudo updatedb; locate Catmandu.pm /usr/share/perl5/Catmandu.pm % grep VERSION /usr/share/perl5/Catmandu.pm our $VERSION = '1.0002'; Everything looks... more or less ok -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #20 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Jonathan Druart from comment #19)
Everything looks... more or less ok
% perl misc/search_tools/rebuild_elastic_search.pl -d -a -b -v misc/search_tools/rebuild_elastic_search.pl: Perl lib version (5.22.2) doesn't match executable '/usr/bin/perl' version (5.24.1) at /usr/lib/i386-linux-gnu/perl/5.22/Config.pm line 62. more or less I am using debian testing. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #21 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- So, I have completely exploded my lxc under testing because of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=836991 (that's the game) Tried again with kohadevbox (so jessie) and I got the same errors http://pastebin.com/JYSVGfGM -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #22 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I seems better with using the debs from deb https://apt.abunchofthings.net/debian jessie main So: % sudo apt install libcatmandu-perl libcatmandu-perl : Depends: libdata-uuid-perl (>= 1.217) % sudo apt install libdata-uuid-perl % sudo apt install libcatmandu-perl % sudo apt install libcatmandu-store-elasticsearch-perl rebuild_elastic_search will raise "Can't locate Catmandu/Importer/MARC.pm" % sudo apt install libcatmandu-marc-perl libcatmandu-marc-perl : Depends: libmarc-xml-perl (>= 1.0.3) but 1.0.2-1kohadev1 is to be installed Because it takes the one from the Koha repo, specify you want the one from Jessie: % sudo apt install -t jessie libmarc-xml-perl % sudo apt install libcatmandu-marc-perl Here you go! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #23 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Catmandu is 1.0002 Search::Elasticsearch is 2.03 Catmandu::Store::ElasticSearch is 0.0501 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #24 from Mirko Tietgen <mirko@abunchofthings.net> --- (In reply to Chris Cormack from comment #12)
* We had to change Catmandu::Store::ElasticSearch::Bag https://github.com/LibreCat/Catmandu-Store-Elasticsearch/issues/10
Is that maybe related to ES claiming all underscore prefix names? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #25 from Nick Clemens <nick@bywatersolutions.com> --- I finally got ES2 to index and return results in kohadevbox, I found I had to do a few things: I used the two .debs attached here and also installed Catmandu from CPAN In Koha/SearchEngine/Elasticsearch.pm I added the line below just before the return: + push @rules, "move_field(_id,my_id)"; In koha-conf I added at the bottom of the elasticsearch stanza: <key_prefix>my_</key_prefix> In koha/SearchEngine/Elasticsearch/Indexer.pm - Catmandu::Importer::MARC->new( records => $records, _id => '999c' ); + Catmandu::Importer::MARC->new( records => $records, id => '999c' ); -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #26 from Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> --- Created attachment 58690 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58690&action=edit Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip -Changed deprecated facets to aggregations -Fixed boolean datatypes not allowing analyzers to be specified -Fixed deprecated '_id' to 'es_id'. Now the ES-index has the correct id==biblionumber ZE TEST PLAN 1. Reset Zebra index since facets are hard coded to dynamic search_marc_mappings. 2. perl misc/search_tools/rebuild_elastic_search.pl 3. Fetch all indexed records and the facet for subject__facet curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{ "aggregations": { "my_agg": { "terms": { "field": "subject__facet" } } } }' -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |olli-antti.kivilahti@jns.fi --- Comment #27 from Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> --- Has there been any discussion about migrating to ElasticSearch 5.1 instead of 2.4? I ran a small test setup back home (using Ubuntu16.04) during the weekends and discovered that I had accidentally installed ElasticSearch 5.1 and Catmandu dependencies from cpan, and that indexing and searching and deleting worked with no probs using the Catmandu::Store::Elasticsearch perldoc examples with "the" compatibility patch in the perldoc. Will look into that tomorrow and hopefully provide a patch for running with ES 5.1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #28 from Chris Cormack <chris@bigballofwax.co.nz> --- (In reply to Olli-Antti Kivilahti from comment #27)
Has there been any discussion about migrating to ElasticSearch 5.1 instead of 2.4?
I ran a small test setup back home (using Ubuntu16.04) during the weekends and discovered that I had accidentally installed ElasticSearch 5.1 and Catmandu dependencies from cpan, and that indexing and searching and deleting worked with no probs using the Catmandu::Store::Elasticsearch perldoc examples with "the" compatibility patch in the perldoc.
Will look into that tomorrow and hopefully provide a patch for running with ES 5.1
Nope no discussion yet (5.1 only came out in December) But I think going to 5.1 seems sensible enough to me -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #29 from Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> --- Created attachment 58757 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=58757&action=edit Bug 17255 - Upgrade Elastic Search code to work with version 5.1 Builds on top of commit: Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip -Fix data type 'string' to 'keyword' and 'text' -index: not_analyzed deprecated, replaced with type: keyword which is equivalent -store: yes was deprecated, use store: true TODO: Installer bindings to both the debian package install and the raw developer install. A taster in Buug 17851 ZE TEST PLAN 0. Remove existing ES and reinstall ES 5.1 apt-get purge elasticsearch Follow instruction here: https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html 1. Reset Elasticsearch index since facets are hard coded to dynamic search_marc_mappings. 1a. perl -e 'use Koha::SearchEngine::Elasticsearch; Koha::SearchEngine::Elasticsearch->reset_elasticsearch_mappings();' 1b. If you get trouble, simply DELETE FROM [search_fields|search_marc_to_fields|search_marc_mapping]; and retry 1a. 1c. Destroy elasticsearch index curl -XDELETE localhost:9200/koha_biblios so it can be recreated 2. Recreate the index: perl misc/search_tools/rebuild_elastic_search.pl 2a. Add something to index if your koha.biblio-table is empty 3. Fetch all indexed records and the facet for subject__facet curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{ "aggregations": { "my_agg": { "terms": { "field": "subject__facet" } } } }' 4. Run the included tests: perl t/db_dependent/Koha_Elasticsearch_Indexer.t perl t/db_dependent/Koha_Elasticsearch.t perl t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t Have fun with your new ES 5.1 cluster! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Upgrade Elastic Search code |Upgrade Elastic Search code |to work with version 2.4+ |to work with version 5.1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #30 from Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> --- Tested on Ubuntu16.04 with: -ES from their own apt-repo. -cpanm Catmandu::Store::Elasticsearch intra search works, facets work, adding records works, full reindex works -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |17885 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17885 [Bug 17885] Koha::SearchEngine::Elasticsearch->reset_elasticsearch_mappings throws DBD::mysql Duplicate entry exceptions -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58690|0 |1 is obsolete| | --- Comment #31 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 59493 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=59493&action=edit Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip -Changed deprecated facets to aggregations -Fixed boolean datatypes not allowing analyzers to be specified -Fixed deprecated '_id' to 'es_id'. Now the ES-index has the correct id==biblionumber ZE TEST PLAN 1. Reset Zebra index since facets are hard coded to dynamic search_marc_mappings. 2. perl misc/search_tools/rebuild_elastic_search.pl 3. Fetch all indexed records and the facet for subject__facet curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{ "aggregations": { "my_agg": { "terms": { "field": "subject__facet" } } } }' Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #58757|0 |1 is obsolete| | --- Comment #32 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 59494 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=59494&action=edit Bug 17255 - Upgrade Elastic Search code to work with version 5.1 Builds on top of commit: Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip -Fix data type 'string' to 'keyword' and 'text' -index: not_analyzed deprecated, replaced with type: keyword which is equivalent -store: yes was deprecated, use store: true TODO: Installer bindings to both the debian package install and the raw developer install. A taster in Buug 17851 ZE TEST PLAN 0. Remove existing ES and reinstall ES 5.1 apt-get purge elasticsearch Follow instruction here: https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html 1. Reset Elasticsearch index since facets are hard coded to dynamic search_marc_mappings. 1a. perl -e 'use Koha::SearchEngine::Elasticsearch; Koha::SearchEngine::Elasticsearch->reset_elasticsearch_mappings();' 1b. If you get trouble, simply DELETE FROM [search_fields|search_marc_to_fields|search_marc_mapping]; and retry 1a. 1c. Destroy elasticsearch index curl -XDELETE localhost:9200/koha_biblios so it can be recreated 2. Recreate the index: perl misc/search_tools/rebuild_elastic_search.pl 2a. Add something to index if your koha.biblio-table is empty 3. Fetch all indexed records and the facet for subject__facet curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{ "aggregations": { "my_agg": { "terms": { "field": "subject__facet" } } } }' 4. Run the included tests: perl t/db_dependent/Koha_Elasticsearch_Indexer.t perl t/db_dependent/Koha_Elasticsearch.t perl t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t Have fun with your new ES 5.1 cluster! Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #33 from Nick Clemens <nick@bywatersolutions.com> --- I had to apply the patches interactively, one at a time, but they worked then without issue. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com QA Contact|testopia@bugs.koha-communit |tomascohen@gmail.com |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|normal |enhancement -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #59493|0 |1 is obsolete| | --- Comment #34 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 59938 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=59938&action=edit Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip -Changed deprecated facets to aggregations -Fixed boolean datatypes not allowing analyzers to be specified -Fixed deprecated '_id' to 'es_id'. Now the ES-index has the correct id==biblionumber ZE TEST PLAN 1. Reset Zebra index since facets are hard coded to dynamic search_marc_mappings. 2. perl misc/search_tools/rebuild_elastic_search.pl 3. Fetch all indexed records and the facet for subject__facet curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{ "aggregations": { "my_agg": { "terms": { "field": "subject__facet" } } } }' Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #59494|0 |1 is obsolete| | --- Comment #35 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 59939 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=59939&action=edit Bug 17255 - Upgrade Elastic Search code to work with version 5.1 Builds on top of commit: Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip -Fix data type 'string' to 'keyword' and 'text' -index: not_analyzed deprecated, replaced with type: keyword which is equivalent -store: yes was deprecated, use store: true TODO: Installer bindings to both the debian package install and the raw developer install. A taster in Buug 17851 ZE TEST PLAN 0. Remove existing ES and reinstall ES 5.1 apt-get purge elasticsearch Follow instruction here: https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html 1. Reset Elasticsearch index since facets are hard coded to dynamic search_marc_mappings. 1a. perl -e 'use Koha::SearchEngine::Elasticsearch; Koha::SearchEngine::Elasticsearch->reset_elasticsearch_mappings();' 1b. If you get trouble, simply DELETE FROM [search_fields|search_marc_to_fields|search_marc_mapping]; and retry 1a. 1c. Destroy elasticsearch index curl -XDELETE localhost:9200/koha_biblios so it can be recreated 2. Recreate the index: perl misc/search_tools/rebuild_elastic_search.pl 2a. Add something to index if your koha.biblio-table is empty 3. Fetch all indexed records and the facet for subject__facet curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{ "aggregations": { "my_agg": { "terms": { "field": "subject__facet" } } } }' 4. Run the included tests: perl t/db_dependent/Koha_Elasticsearch_Indexer.t perl t/db_dependent/Koha_Elasticsearch.t perl t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t Have fun with your new ES 5.1 cluster! Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|srdjan@catalyst.net.nz |olli-antti.kivilahti@jns.fi -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master CC| |kyle@bywatersolutions.com --- Comment #36 from Kyle M Hall <kyle@bywatersolutions.com> --- Pushed to master for 17.05, thanks Olli! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|Pushed to Master |RESOLVED CC| |katrin.fischer@bsz-bw.de --- Comment #37 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- This won't get ported back to 16.11.x as it is an enhancement. But: yay! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |18085 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18085 [Bug 18085] Elasticsearch ids are not being stored correctly -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 --- Comment #38 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Authority search seems broken, someone faced it? [Request] ** [http://localhost:9200]-[400] [illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [Heading.phrase] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory., called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/local/share/perl/5.20.2/Catmandu/Store/ElasticSearch/Bag.pm line 159. With vars: {'request' => {'serialize' => 'std','method' => 'GET','body' => {'sort' => [{'Heading.phrase' => 'asc'}],'from' => 0,'size' => 20,'query' => {'bool' => {'should' => [{'match' => {'Heading-Main' => ''}}]}}},'ignore' => [],'mime_type' => 'application/json','path' => '/koha_kohadev_authorities/data/_search','qs' => {}},'status_code' => 400,'body' => {'status' => 400,'error' => {'caused_by' => {'reason' => 'Fielddata is disabled on text fields by default. Set fielddata=true on [Heading.phrase] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.','type' => 'illegal_argument_exception'},'phase' => 'query','grouped' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),'failed_shards' => [{'shard' => 0,'node' => 'or12QGCqT2mEEFuFMr4wFA','index' => 'koha_kohadev_authorities','reason' => {'reason' => 'Fielddata is disabled on text fields by default. Set fielddata=true on [Heading.phrase] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.','type' => 'illegal_argument_exception'}}],'type' => 'search_phase_execution_exception','root_cause' => [{'type' => 'illegal_argument_exception','reason' => 'Fielddata is disabled on text fields by default. Set fielddata=true on [Heading.phrase] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.'}],'reason' => 'all shards failed'}}} Unable to perform your search. Please try again. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17255 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |18189 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18189 [Bug 18189] Elasticsearch sorting broken -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org