https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 --- Comment #2 from Julian Maurice <julian.maurice@biblibre.com> --- Created attachment 148850 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148850&action=edit Bug 33353: Add compatibility with Search::Elasticsearch 8.0 For some reasons, with Search::Elasticsearch 8.0 JSON::true is interpreted as 1 and Elasticsearch will always respond that it found "at least 1 record". Something like this: { total => { value => 1, relation => 'gte' } } Replacing JSON::true by \1 works with every version of Search::Elasticsearch I tested (6.80, 7.717, 8.0) Also worth noting: Search::Elasticsearch will stop being supported by Elastic https://github.com/elastic/elasticsearch-perl/issues/220 We might need to find an alternative for future versions of Elasticsearch -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |victor@tuxayo.net --- Comment #3 from Victor Grousset/tuxayo <victor@tuxayo.net> --- lol replacing JSON::true by the magic number 1, weird change that ES are doing. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #148850|0 |1 is obsolete| | --- Comment #5 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Created attachment 148938 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148938&action=edit Bug 33353: Add compatibility with Search::Elasticsearch 8.0 For some reasons, with Search::Elasticsearch 8.0 JSON::true is interpreted as 1 and Elasticsearch will always respond that it found "at least 1 record". Something like this: { total => { value => 1, relation => 'gte' } } Replacing JSON::true by \1 works with every version of Search::Elasticsearch I tested (6.80, 7.717, 8.0) Also worth noting: Search::Elasticsearch will stop being supported by Elastic https://github.com/elastic/elasticsearch-perl/issues/220 We might need to find an alternative for future versions of Elasticsearch Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 --- Comment #7 from Victor Grousset/tuxayo <victor@tuxayo.net> --- By the way, these patches mean end of compatibility with ES6. Not a problem in my opinion, ES6 is EOL since 10 Feb 2022. And now that we have OpenSearch 1.x compat (still need to officialize it but it's there) people that are bothered with the new ES license (since ES 7.11) not really being libre/open source anymore (it's a mess, very controversial and blurry) can use OpenSearch 1.x -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=25439 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 Philip Orr <philip.orr@lmscloud.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #148938|0 |1 is obsolete| | --- Comment #9 from Philip Orr <philip.orr@lmscloud.de> --- Created attachment 148989 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148989&action=edit Bug 33353: Add compatibility with Search::Elasticsearch 8.0 For some reasons, with Search::Elasticsearch 8.0 JSON::true is interpreted as 1 and Elasticsearch will always respond that it found "at least 1 record". Something like this: { total => { value => 1, relation => 'gte' } } Replacing JSON::true by \1 works with every version of Search::Elasticsearch I tested (6.80, 7.717, 8.0) Also worth noting: Search::Elasticsearch will stop being supported by Elastic https://github.com/elastic/elasticsearch-perl/issues/220 We might need to find an alternative for future versions of Elasticsearch Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Philip Orr <philip.orr@lmscloud.de> -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 --- Comment #10 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- What documentation do we need to update once this is pushed? Wiki, website,...? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA --- Comment #12 from Victor Grousset/tuxayo <victor@tuxayo.net> --- By the way, thanks Philip for the test. It counts as a signoff and so I can QA! A second signoff is always useful. Works, makes sense, QA script happy, code looks good, passing QA :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 David Liddle <david.liddle@wycliff.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david.liddle@wycliff.de -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 --- Comment #14 from David Liddle <david.liddle@wycliff.de> --- (In reply to Victor Grousset/tuxayo from comment #13)

After having this in main/master there will be the question of backporting (now or later) to 22.11 which is the LTS. Because before the end of the LTS it's likely that ES 7 won't be supported anymore.

Here https://www.elastic.co/support/eol#elasticsearch we can see that branch 7.x will be supported until 9.x comes out.

There is no alpha version of 9.x for now, but when there will be, the stable release can be expected 6 months later. (learned this in ES forums) So people interested can keep an eye here: https://github.com/elastic/elasticsearch/tags for when the 1st 9.x alpha pops.

Also, for OpenSearch, it's already planned that 1.x will reach end of life before the end of 22.11's support. https://opensearch.org/releases.html => "Maintenance Window End" => "December 31, 2023"

So what's the best strategy? (because it implies dropping 6.x support) 1. backport now 2. backport when/a bit before OpenSearch 1.x support ends (December 31, 2023) 3. backport when/a bit before ElasticSearch 7.x support will have also ended. (we don't know yet, it could be sooner than OS 1 or later) 4. don't backport and require ES 7 or OS 1 Koha admins to keep an end of life search engine or upgrade from the LTS.

This bug caught my attention last week, and I have been watching with interest to see what responses there would be to Victor's question here. In the absence of any (so far), I would like to offer the perspective of a system administrator in the wild: - The cataloger for our consortium of small libraries has been itching to implement Elasticsearch for more than a year. - In examining the request, I was disconcerted by a few facts about Elasticsearch: 1. The version included by Koha appeared to be one that had reached end-of-life with the publisher, 2. Even the supported version had been associated with significant data breaches, and 3. There seemed to be a lack of clarity following the change in licensing and the open source world's response to it. - Given the above, there was no scenario in this corner of the multiverse in which I would have installed Elasticsearch on the internet-facing servers for which I was responsible. - HOWEVER, seeing OpenSearch on the horizon creates a new situation. Being well-supported by AWS, it has a chance of being a viable and safe alternative. - Our production server runs 22.05, BUT I can imagine a willingness to leap forward a bit if OpenSearch is backported to 22.11. - If that is indeed the future of more advanced indexing and searching, then it seems logical (to me) to backport it to an LTS release, if you have the resources to do so. Perhaps the question needs the attention of a broader or more representative forum, if you have one. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 --- Comment #15 from Victor Grousset/tuxayo <victor@tuxayo.net> ---

This bug caught my attention last week, and I have been watching with interest to see what responses there would be to Victor's question here. In the absence of any (so far), I would like to offer the perspective of a system administrator in the wild:

Hi, welcome :) More input can definitely help to unblock this.

1. The version included by Koha appeared to be one that had reached end-of-life with the publisher

ES 7 and OpenSearch 1.X (which isn't eol) has been supported since a year. But we didn't update the systems requirements documentation (now it's finally updated) due to not having the time to search for real work usage to confidently claim support :-/

2. Even the supported version had been associated with significant data breaches

You mean the log4j vulnerabilities? (latest 6.x patched them) or like bad defaults? Is your statement still valid for ES 7.x?

3. There seemed to be a lack of clarity following the change in licensing and the open source world's response to it.

Indeed! About that, to have more material to raise awareness about the issue, do you happen to know more about the open source world's response to it? Besides Debian, Fedora and the Open Source Initiative not considering the SSPL license libre/open source?

- Our production server runs 22.05, BUT I can imagine a willingness to leap forward a bit if OpenSearch is backported to 22.11.

You can do that right away with OpenSearch 1.x , 22.05 supported it on launch. Sorry again for the delay in advertising it's support. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the| |These changes to support release notes| |ElasticSearch 8.x and | |OpenSearch 2.x come with a | |loss of support for | |ElasticSearch 6.x. | |Existing | |instances will have to | |upgrade to either | |ElasticSearch 7.x or 8.x or | |OpenSearch 1.x or 2.x -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 --- Comment #19 from Victor Grousset/tuxayo <victor@tuxayo.net> --- (In reply to Nick Clemens from comment #17)

Making the upgrade of Koha force the ES upgrade without allowing it before feels like it could cause issues

Thanks for catching that 😱 that's a big miss in my review. Requiring reindex wouldn't make the upgrade as simple as hoped. How can one reproduce an ES upgrade with KTD? That would mean shutting down the ES7 container and restart an ES8 one without loosing the existing data. https://www.elastic.co/guide/en/cloud/current/ec-migrating-data.html Maybe something with a snapshot and moving the snapshot files out of the container and putting then in the ES8 container? Or starting a parallel ES8 container, having it replicate the data from the ES7 one and replace the ES7 with an ES8 (to have the right container name to plug in KTD) and replicate again? (In reply to Nick Clemens from comment #18)

I believe you can use es directly to update and remove types and avoid reindex:

curl -X POST http://es:9200/_reindex -H 'Content-Type: application/json' -d '{"source":{"index": "koha_kohadev_biblios","type": "data"},"dest":{"index": "koha_kohadev_biblio_s"}}'

that reindexes ES internally to remove the types

So it's much faster than reindexing from Koha's data, that's the gain, right?

- it does require renaming the index so would need some sort of alias or something

Does doing the rename again would allow to get back to original name and have no config to touch? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 --- Comment #20 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Victor Grousset/tuxayo from comment #19)

(In reply to Nick Clemens from comment #17)

...

Making the upgrade of Koha force the ES upgrade without allowing it before feels like it could cause issues

Thanks for catching that 😱 that's a big miss in my review. Requiring reindex wouldn't make the upgrade as simple as hoped.

How can one reproduce an ES upgrade with KTD? That would mean shutting down the ES7 container and restart an ES8 one without loosing the existing data. https://www.elastic.co/guide/en/cloud/current/ec-migrating-data.html Maybe something with a snapshot and moving the snapshot files out of the container and putting then in the ES8 container? Or starting a parallel ES8 container, having it replicate the data from the ES7 one and replace the ES7 with an ES8 (to have the right container name to plug in KTD) and replicate again?

That would be tricky - maybe if you setup a volume, and then just connected new container to the volume? Or get into the container and perform an upgrade directly? I think it should work for testing to spin up on ES7 - apply these patches, ES won't work, then do the reindex (as below or via Koha) and confirm that it works. We aren't removing 7 support afaik, but require 7 or higher after this I believe

(In reply to Nick Clemens from comment #18)

...

I believe you can use es directly to update and remove types and avoid reindex: So it's much faster than reindexing from Koha's data, that's the gain, right?

Yes, as I understand it moves the data to remove the type level, so we don't have to cycle through every record and parse into ES - I tested at Hackfest and it was exponentially faster

...

- it does require renaming the index so would need some sort of alias or something

Does doing the rename again would allow to get back to original name and have no config to touch?

I think you either setup an alias with the old that points to the new name - or do the reindex, delete the old one, then rename the new one -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 --- Comment #21 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 23.11. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 --- Comment #23 from Victor Grousset/tuxayo <victor@tuxayo.net> --- So as it stands what to know about upgrading, now that it's in main/master is that people will need either do a full reindex. Or an internal reindex, details in comment 18 and the end of comment 20 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- Text to go in the|These changes to support |These changes to support release notes|ElasticSearch 8.x and |ElasticSearch 8.x and |OpenSearch 2.x come with a |OpenSearch 2.x come with a |loss of support for |loss of support for |ElasticSearch 6.x. |ElasticSearch 6.x. |Existing | |instances will have to |Existing instances will |upgrade to either |have to upgrade to either |ElasticSearch 7.x or 8.x or |ElasticSearch 7.x or 8.x or |OpenSearch 1.x or 2.x |OpenSearch 1.x or 2.x | | | |Upgrade from ES 7.x or OS | |1.X to ES 8.x or OS 2.x | |require a reindexation. Summary|Add compatibility with |Add compatibility with |Elasticsearch 8 |Elasticsearch 8 and | |OpenSearch 2 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fridolin.somers@biblibre.co | |m Status|Pushed to master |Needs documenting --- Comment #26 from Fridolin Somers <fridolin.somers@biblibre.com> --- Enhancement not pushed to 23.05.x -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 M <schodkowy.omegi-0r@icloud.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35618 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35618 [Bug 35618] catalogue/showelastic.pl uses deprecated/removed parameter "type" -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 Alexander Borkowski <alexander.borkowski@kau.se> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |alexander.borkowski@kau.se --- Comment #28 from Alexander Borkowski <alexander.borkowski@kau.se> --- ES 7 has reached End of Maintenance Term now (i.e. as of 2025-04-15). While this does not make it EOL, my understanding is that it will no longer receive security updates. We are still on Koha LTS 22.11. Right now the easiest option for me looks to be to backport this so we can upgrade to ES 8. What is required to do this? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33353 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wainuiwitikapark@catalyst.n | |et.nz Version(s)|23.11.00 |23.11.00,22.11.36 released in| | --- Comment #30 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- Applied to 22.11.x for 22.11.36 -- You are receiving this mail because: You are watching all bug changes.