[Bug 39314] New: RabbitMQ can't upgrade from 3.x to 4.x without running a data migration on 3.13
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 Bug ID: 39314 Summary: RabbitMQ can't upgrade from 3.x to 4.x without running a data migration on 3.13 Change sponsored?: --- Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: Installation and upgrade (command-line installer) Assignee: koha-bugs@lists.koha-community.org Reporter: victor@tuxayo.net QA Contact: testopia@bugs.koha-community.org CC: tomascohen@gmail.com Reported to Debian as it affects their next release which will do a jump from 3.10 to 4.0: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1100165#10 Which can't work: https://www.rabbitmq.com/docs/upgrade «You can only upgrade to RabbitMQ 4.0 from RabbitMQ 3.13.» «stable feature flags have to be enabled before the upgrade. The upgrade will fail if you miss this step.» See there more more details. ----- For us: is there still data in RabbitMQ when stopping a Koha instance and upgrading it? I recall reading that the DB was basically mirroring the message queue data during a discussion about the actual need for a dedicated message queue. Does it mean the koha-common package can wipe the RabbitMQ data which will allow it to start without issue? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #1 from David Cook <dcook@prosentient.com.au> --- (In reply to Victor Grousset/tuxayo from comment #0)
Reported to Debian as it affects their next release which will do a jump from 3.10 to 4.0: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1100165#10
Which can't work: https://www.rabbitmq.com/docs/upgrade «You can only upgrade to RabbitMQ 4.0 from RabbitMQ 3.13.»
That's fun...
For us: is there still data in RabbitMQ when stopping a Koha instance and upgrading it? I recall reading that the DB was basically mirroring the message queue data during a discussion about the actual need for a dedicated message queue. Does it mean the koha-common package can wipe the RabbitMQ data which will allow it to start without issue?
The DB stores the jobs, and the Koha app passes messages to background workers via RabbitMQ. If the background worker starts up and can't get a connection to RabbitMQ, it'll query the DB directly for the jobs. What this means is that if you stopped koha-common, nuked RabbitMQ, re-installed RabbitMQ, and started koha-common, you could have data loss if there were any messages in RabbitMQ that hadn't been processed. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- A dedicated sysadmin who knows to look out for the RabbitMQ issue could perform a RabbitMQ reinstall without a problem. But for someone without sysadmin support... there could be problems depending when the upgrade is performed. =- I'm trying to think of workarounds that tried to determine if there was any pending messages and re-queue or otherwise deal with those messages... but it would probably be messy. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #3 from Victor Grousset/tuxayo <victor@tuxayo.net> ---
A dedicated sysadmin who knows to look out for the RabbitMQ issue could perform a RabbitMQ reinstall without a problem
They would have to find a package somewhere for the latest 3.X just for the upgrade. Still quite the hassle >_< -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- (In reply to Victor Grousset/tuxayo from comment #3)
A dedicated sysadmin who knows to look out for the RabbitMQ issue could perform a RabbitMQ reinstall without a problem
They would have to find a package somewhere for the latest 3.X just for the upgrade. Still quite the hassle >_<
Sure, but it's not uncommon. I think Keycloak requires stepped upgrades for certain versions. PostgreSQL too for really old versions. We try to make things seamless for people, but I think there needs to be limits too. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #5 from Jonathan Druart <jonathan.druart@gmail.com> --- *** Bug 39868 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #6 from Jonathan Druart <jonathan.druart@gmail.com> --- There is also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1104149 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 Aleisha Amohia <aleisha@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aleisha@catalyst.net.nz -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #7 from Jonathan Druart <jonathan.druart@gmail.com> --- Is this still valid? https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1104149#10 "I wasn't able to reproduce, so probably it does work now?" -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #8 from Victor Grousset/tuxayo <victor@tuxayo.net> --- It seems these turn out to be different problems. See 39868 bug comment 4 -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #9 from Victor Grousset/tuxayo <victor@tuxayo.net> --- On the Debian side, there isn't much they can do without causing other issues. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1100165#44 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1100165#51 At least there are some strategies:
Heavy RabbitMQ users that cannot reset their /var/lib/rabbitmq/mnesia database could maybe use upstream docker image to attempt a rolling upgrade. Otherwise, users are advised to simply drain all queues and make clients point to a new cluster if that's possible.
At least the Debian release notes should warn about this:
writing about it in the Trixie release notes
I don't know about Ubuntu. And whether that's enough anyway. Should we put a note on the release notes of Bug 41077? Wait, that wouldn't work since there is no patch. Same if we put something in the release notes here. Any ideas? (In reply to David Cook from comment #1)
tuxayo:
For us: is there still data in RabbitMQ when stopping a Koha instance and upgrading it? [...]
[...] What this means is that if you stopped koha-common, nuked RabbitMQ, re-installed RabbitMQ, and started koha-common, you could have data loss if there were any messages in RabbitMQ that hadn't been processed.
Is it something that regularly happens? (have messages in RabbitMQ that haven't been processed) With the reasonable assumption that people don't upgrade Koha in the middle of a day with librarians doing things that use the queue/workers. Or is it something that totally happens? Maybe the assumption is wrong or there is another source of messages that is unavoidable. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=41506 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 Oliver Behnke <oliver.behnke@aei.mpg.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |oliver.behnke@aei.mpg.de --- Comment #10 from Oliver Behnke <oliver.behnke@aei.mpg.de> --- Just ran into this issue myself and resolved it (before I came here) by "simply" installing the intermediate versions required, one by one, enabling their feature flags accordingly. Pretty annoying but I honestly don't see a way around that, presuming that Debian releases will always jump accross a few minor releases of RabbitMQ. It's all outlined in [1] and [2]. That latter also tells you which intermediate releases you need to install: - Look at the first column ("Require") of that table and find the next version higher than your current one *and* where the second column ("Stable") has a version higher than your's - Then check the second column and find the highest version: that's your next target - Install it, enable all features, then repeat For example: 1) Debian bookworm provides 3.10.8 which means 3.12.0 is the next upgrade step that needs flags which your version doesn't yet include. So you want to upgrade to at least 3.11.0 in this case (the latest revision is 3.11.28) to prepare for 3.12 2) Debian trixie comes with 4.0.5 which also happens to be the next upgrade target to prepare for. According to the flags table 3.13.1 is the latest release introducing a new flag for 4.0, so that's what we install (the latest revision is 3.13.28). Note: there's no need to install 3.12 first since 3.13 did *not* have any additional flag requirements. However, to make things worse RabbitMQ releases depend on different Erlang versions. And to make it even more worse, they require a specific major version. E.g. 3.10 depends on Erlang 25, whereas 3.13 depends on Erlang 26, while 4.0 depends on Erlang 27. Yeah, so we also get to mess with those! Fortunately, they provide Debian repos [3] for Erlang and RabbitMQ making things a bit easier. But be sure to pin those repos if you want to stick to Debian's packages by default. Hope this helps [1] https://www.rabbitmq.com/blog/2022/07/20/required-feature-flags-in-rabbitmq-... [2] https://www.rabbitmq.com/docs/feature-flags#core-feature-flags [3] https://www.rabbitmq.com/docs/install-debian#apt-repositories -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |In Discussion -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 davewood@gmx.at changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |davewood@gmx.at --- Comment #11 from davewood@gmx.at --- @Oliver Behnke I think you made a mistake with the rabbitmq versions. There is no version 3.13.28 according to https://www.rabbitmq.com/release-information Latest 3.13.x release is 3.13.7 ps: thanks for the infos you shared! -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #12 from davewood@gmx.at --- i started fixing our broken debian13 rabbitmq-server packages and this list of commands works for us at HKS3 (www-koha-support.eu) to fix the issue. ``` curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A..." | gpg --dearmor | tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null tee /etc/apt/sources.list.d/rabbitmq.list <<EOF deb [arch=amd64 signed-by=/usr/share/keyrings/com.rabbitmq.team.gpg] https://deb1.rabbitmq.com/rabbitmq-erlang/debian/bookworm bookworm main deb [arch=amd64 signed-by=/usr/share/keyrings/com.rabbitmq.team.gpg] https://deb2.rabbitmq.com/rabbitmq-erlang/debian/bookworm bookworm main deb [arch=amd64 signed-by=/usr/share/keyrings/com.rabbitmq.team.gpg] https://deb1.rabbitmq.com/rabbitmq-server/debian/trixie trixie main deb [arch=amd64 signed-by=/usr/share/keyrings/com.rabbitmq.team.gpg] https://deb2.rabbitmq.com/rabbitmq-server/debian/trixie trixie main EOF apt update apt install rabbitmq-server=3.11.28-1 erlang-base:amd64=1:25.3.2.21-1 erlang-crypto:amd64=1:25.3.2.21-1 erlang-ssl:amd64=1:25.3.2.21-1 erlang-public-key:amd64=1:25.3.2.21-1 erlang-asn1:amd64=1:25.3.2.21-1 erlang-runtime-tools:amd64=1:25.3.2.21-1 erlang-mnesia:amd64=1:25.3.2.21-1 erlang-eldap:amd64=1:25.3.2.21-1 erlang-tools:amd64=1:25.3.2.21-1 erlang-os-mon:amd64=1:25.3.2.21-1 erlang-snmp:amd64=1:25.3.2.21-1 erlang-inets:amd64=1:25.3.2.21-1 erlang-ftp:amd64=1:25.3.2.21-1 erlang-tftp:amd64=1:25.3.2.21-1 erlang-parsetools:amd64=1:25.3.2.21-1 erlang-xmerl:amd64=1:25.3.2.21-1 erlang-syntax-tools:amd64=1:25.3.2.21-1 # optional if last step fails with {case_clause,version_not_available} error rm -rf /var/lib/rabbitmq/mnesia/rabbit* systemctl restart rabbitmq-server.service rabbitmqctl enable_feature_flag all apt install rabbitmq-server=3.13.7-1 erlang-base:amd64=1:26.2.5.14-1 erlang-xmerl:amd64=1:26.2.5.14-1 erlang-os-mon:amd64=1:26.2.5.14-1 erlang-mnesia:amd64=1:26.2.5.14-1 erlang-snmp:amd64=1:26.2.5.14-1 erlang-crypto:amd64=1:26.2.5.14-1 erlang-runtime-tools:amd64=1:26.2.5.14-1 erlang-ssl:amd64=1:26.2.5.14-1 erlang-public-key:amd64=1:26.2.5.14-1 erlang-asn1:amd64=1:26.2.5.14-1 erlang-eldap:amd64=1:26.2.5.14-1 erlang-inets:amd64=1:26.2.5.14-1 erlang-ftp:amd64=1:26.2.5.14-1 erlang-tftp:amd64=1:26.2.5.14-1 erlang-tools:amd64=1:26.2.5.14-1 erlang-syntax-tools:amd64=1:26.2.5.14-1 erlang-parsetools:amd64=1:26.2.5.14-1 systemctl restart rabbitmq-server.service rabbitmqctl enable_feature_flag all rm /etc/apt/sources.list.d/rabbitmq.list rm /usr/share/keyrings/com.rabbitmq.team.gpg apt update apt upgrade systemctl restart rabbitmq-server.service rabbitmqctl enable_feature_flag all ``` -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #13 from Oliver Behnke <oliver.behnke@aei.mpg.de> --- (In reply to davewood from comment #11)
I think you made a mistake with the rabbitmq versions.
There is no version 3.13.28 according to https://www.rabbitmq.com/release-information
Latest 3.13.x release is 3.13.7
Oops, you're right. I probably confused the patch level with 3.11.28 (above). Sorry.
ps: thanks for the infos you shared!
Glad they helped. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #14 from davewood@gmx.at --- im afraid i've encountered the same problem again. upgrading koha-common (from 26.05.02-1 to 26.05.03-1) on debian 13.6 fails. ``` ob for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xeu rabbitmq-server.service" for details. dpkg: error processing package koha-common (--configure): installed koha-common package post-installation script subprocess returned error exit status 1 Errors were encountered while processing: koha-common Error: Sub-process /usr/bin/dpkg returned an error code (1) ``` ``` systemctl status rabbitmq-server.service ● rabbitmq-server.service - RabbitMQ Messaging Server Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; disabled; preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Wed 2026-09-02 09:54:01 UTC; 1s ago Invocation: 43e934516b1448d7ba5e73bf93232b64 Process: 7497 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=exited, status=1/FAILURE) Main PID: 7497 (code=exited, status=1/FAILURE) Status: "Standing by" Mem peak: 76.4M CPU: 2.821s ``` ``` journalctl -xeu rabbitmq-server.service ░░ Support: https://www.debian.org/support ░░ ░░ A start job for unit rabbitmq-server.service has begun execution. ░░ ░░ The job identifier is 2942. Sep 02 09:54:50 koha-maxlib systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=1/FAILURE ░░ Subject: Unit process exited ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ An ExecStart= process belonging to unit rabbitmq-server.service has exited. ░░ ░░ The process' exit code is 'exited' and its exit status is 1. Sep 02 09:54:50 koha-maxlib systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'. ░░ Subject: Unit failed ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ The unit rabbitmq-server.service has entered the 'failed' state with result 'exit-code'. Sep 02 09:54:50 koha-maxlib systemd[1]: Failed to start rabbitmq-server.service - RabbitMQ Messaging Server. ░░ Subject: A start job for unit rabbitmq-server.service has failed ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ A start job for unit rabbitmq-server.service has finished with a failure. ░░ ░░ The job identifier is 2942 and the job result is failed. Sep 02 09:54:50 koha-maxlib systemd[1]: rabbitmq-server.service: Consumed 2.694s CPU time, 80.6M memory peak. ░░ Subject: Resources consumed by unit runtime ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ The unit rabbitmq-server.service completed and consumed the indicated resources. Sep 02 09:55:00 koha-maxlib systemd[1]: rabbitmq-server.service: Scheduled restart job, restart counter is at 2. ░░ Subject: Automatic restarting of a unit has been scheduled ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ Automatic restarting of the unit rabbitmq-server.service has been scheduled, as the result for ░░ the configured Restart= setting for the unit. Sep 02 09:55:00 koha-maxlib systemd[1]: Starting rabbitmq-server.service - RabbitMQ Messaging Server... ░░ Subject: A start job for unit rabbitmq-server.service has begun execution ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ A start job for unit rabbitmq-server.service has begun execution. ░░ ░░ The job identifier is 3002. ``` -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #15 from Oliver Behnke <oliver.behnke@aei.mpg.de> --- Is this an FYI or do you help fixing it, e.g. doing the incremental updates as outlined in bug 39314 comment 10? -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #16 from Oliver Behnke <oliver.behnke@aei.mpg.de> --- Missing "need". Sorry, but bugzilla sucks (not being able to edit own comments). -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314 --- Comment #17 from davewood@gmx.at --- this sequence fixes the latest issue for us. it purges the rabbitmq though. so only use if you use DB polling. and for good measure, make a snapshot or backup or whatever you need to roll back in case of an error. no warranty. ``` systemctl stop rabbitmq-server.service mv "/var/lib/rabbitmq/mnesia" "var_lib_rabbitmq_mnesia" rm -f "var/lib/rabbitmq/erl_crash.dump" install -d -o rabbitmq -g rabbitmq -m 0750 "/var/lib/rabbitmq/mnesia" systemctl start rabbitmq-server.service systemctl reset-failed rabbitmq-server.service rabbitmq-plugins enable --offline rabbitmq_stomp systemctl restart rabbitmq-server.service rabbitmqctl enable_feature_flag all dpkg --configure koha-common apt update && apt upgrade -y apt autoclean && apt autoremove systemctl stop rabbitmq-server.service ``` -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org