https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #1 from Tristin Stagg <tristin.stagg@bywatersolutions.com> --- Currently, the koha-common.service systemd unit is just a wrapper around /etc/init.d/koha-common, which internally uses Koha’s shell wrappers and the daemon command to launch all Koha subcomponents (Plack, Zebra, Z39.50 responder, SIP2 server, background workers, indexers, etc.) under a single systemd service. This has several problems: 1) No process isolation: All subcomponents for all Koha instances are children of koha-common.service. 2) OOM-killer issues: When the system is under memory pressure, the Linux OOM killer can indiscriminately kill any Koha sub-daemon. In our environment, we’ve seen background workers or Plack workers being killed silently. 3) Poor observability: systemctl status koha-common shows a giant mixed tree of processes. It is difficult to identify which component is misbehaving or using excessive resources. 4) systemd integration is broken: Features like per-service cgroup limits (MemoryMax=, CPUQuota=), restart policies, logging (journalctl -u), and failure detection cannot be used at the component level. Environment: - Debian 10, 11, 12 - Koha version: At least 24.11-25.05 Steps to Reproduce: 1) Install Koha via Debian packages. 2) Start an instance, e.g. koha-create --create-db mainlib. 3) Run systemctl status koha-common.service. 4) Observe that all daemons (Plack, Zebra, workers, indexers, Z39.50, etc.) run under the single koha-common.service process tree. Observed Behavior: All Koha daemons for all instances run under one service. For example: ● koha-common.service - Start required services for each Koha instance Loaded: loaded (/etc/init.d/koha-common; enabled) Active: active (running) CGroup: /system.slice/koha-common.service ├─ starman master … ├─ starman worker … ├─ zebra … ├─ background_jobs_worker.pl … ├─ es_indexer_daemon.pl … └─ etc. When memory pressure occurs, OOM-killer may kill any of these, and systemd does not restart them because they are just sub-processes of koha-common. Expected Behavior: Each Koha subcomponent (Plack, Zebra, workers, indexers, Z39.50, SIP2, etc.) should run as its own systemd unit, ideally with instance scoping. For example: - koha-plack@instance.service - koha-zebra@instance.service - koha-z3950@instance.service - koha-workers@instance.service - koha-indexer@instance.service And koha-common.service could be a meta unit (or target) that pulls in all the per-instance units. This would allow: - Proper systemd supervision of each daemon - Restart policies if one crashes - Independent resource limits - Better logging and observability Why this matters: Right now, Koha administrators lose visibility and control of critical services. In production environments, a single rogue process can consume memory, trigger the OOM-killer, and kill unrelated Koha daemons, taking down parts of the system silently. This design makes Koha fragile on modern Linux systems. Suggested Fix: 1) Replace the init.d style management with native systemd unit files for each Koha component. 2) Use templated units for instance support (koha-plack@.service, etc.). 3) Have koha-common.service or koha.target depend on these per-instance units. 4) This would align Koha with best practices for systemd-based services and improve stability, observability, and maintainability in production. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Nick Clemens (kidclamp) <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au, | |jonathan.druart@gmail.com, | |julian.maurice@biblibre.com | |, m.de.rooy@rijksmuseum.nl, | |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #2 from David Cook <dcook@prosentient.com.au> --- (In reply to Tristin Stagg from comment #1) Thanks, Tristin, for all that info. I agree with a lot of what you're saying here. That said, I think there are some challenges with it. Generally speaking, SystemD isn't available in Docker containers, so this isn't going to work for koha-testing-docker. If we were to more fully embrace SystemD, we'd need to have viable alternatives for koha-testing-docker to function. But if we did have alternatives for koha-testing-docker to function, it wouldn't really be testing Koha as much as we are now. I don't think that's necessarily a blocker, but it's something that would need to be carefully considered.

Suggested Fix:

1) Replace the init.d style management with native systemd unit files for each Koha component. 2) Use templated units for instance support (koha-plack@.service, etc.). 3) Have koha-common.service or koha.target depend on these per-instance units. 4) This would align Koha with best practices for systemd-based services and improve stability, observability, and maintainability in production.

I have a different system I deploy using Debian packages and I have "debian/pkg.app.service" and "debian/pkg.app@.service" files where "debian/pkg.app.service" depends on "app@first.service app@second.service". It works well, but it is also built into the package itself which lives in the git repo. I am curious how we'd use SystemD templates with Koha's dynamic instance management with that dependency in mind. This might be possible and I'm just not familiar enough with unit files to know how to do it. It's just a concern that comes to mind at the moment. (After doing a bit of Googling, it looks like it might be possible to provide a "foo.service.wants/" directory. I suppose a person might also use an *override.conf file. I hadn't thought of that before although I've used those in other places for other reasons.) -- So overall... I think it sounds good. The only sticking point I can think of is koha-testing-docker. But maybe that will become clearer as we re-tool the SystemD files. Interesting idea in any case! -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff Patch complexity|--- |Medium patch -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #5 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 195098 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=195098&action=edit Bug 40901: Add koha-sysv package for SysV init support This patch introduces the koha-sysv package, which provides traditional SysV init.d scripts for managing Koha instances. All services (Plack, Zebra, SIP, Z3950, workers, indexers) are managed through a single init script using the daemon wrapper. The package includes: - /etc/init.d/koha-common (SysV init script) - /lib/systemd/system/koha-common.service (systemd wrapper for init.d) This package is intended for legacy systems, containers without systemd (like KTD), or environments where SysV init is preferred. The package provides 'koha-init' and conflicts with koha-systemd to ensure only one init system is active at a time. Test plan: 1. Build and install the package 2. Start services with /etc/init.d/koha-common start 3. Verify all services start correctly 4. Check that koha-systemd cannot be installed alongside -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #7 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Ok, I gave this a go. To summarize: I introduced a new package `koha-systemd` with the result of my experimentation on how to handle per-instance Koha service units. It works like a charm on its own. Please refer to a separate project I made for illustration purposes [1]. If you have a running test server in which systemd is available, just follow the readme and test it yourself. It is just installing koha-systemd, disabling koha-common and following the instructions. That said, I then put it together here for upstream inclusion and I made the call to keep 'the old init' for the sake of having a lean path for introducing this change without breaking every single CI and workflow the community has. The approach is: 1) Make `koha-common` a meta-package that depends on `koha-init`. The latter is a virtual package provided by either `koha-systemd` or `koha-sysv` in that order. If `systemd` is not available, the `koha-systemd` is not installable so it falls back to `koha-sysv`. 2) `koha-common` now also depends on `koha-core` which pulls all the other deps. 3) KTD could do `apt install koha-sysv koha-core` instead of `koha-common`. We'll see later. Please analyze the proposed implementation. Take the time to test the package I created in the external repository to ease testing. TODO: All `koha-*` scripts that deal with daemons e.g. `koha-plack`, `koha-zebra`, etc need to be added a detection mechanism so they do the right thing on each context. It should be straight-forward to implement too. Will submit if there's consensus. [1] https://github.com/thekesolutions/koha-systemd -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi (tcohen) from comment #7)

That said, I then put it together here for upstream inclusion and I made the call to keep 'the old init' for the sake of having a lean path for introducing this change without breaking every single CI and workflow the community has.

It looks pretty interesting overall. I'm glad that you did keep a koha-sysv package. As you say, we'll still need that for the koha-testing-docker image. It'll be interesting updating the /usr/sbin/koha-* scripts. I'm not sure that it will be straightforward, but fingers crossed. How would we intend to test the systemd stuff on an on-going basis? Using a privileged container in the CI? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi (tcohen) from comment #9)

...

How would we intend to test the systemd stuff on an on-going basis? Using a privileged container in the CI?

I've never run SystemD in a container and am unsure that's possible. I'd just spin up a VPS and just try a regular install with the extra package installed. I mean, I wouldn't bother more than that heh.

I'm sure that I've done it, but it was a while ago. Red Hat provide a ubi-init image which uses SystemD but I'm trying to remember if it just works with podman. I think maybe it works with docker but is more difficult. In my case, I end up using SysV in a Docker dev env and SystemD in non-Docker test/prod envs. In this case though I was wondering what the Koha community would do? -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #12 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- This is looking great.. the infra guys here said good things (I shan't repeat as they were not polite/appropriate for public consumption ;) ) Needless to say, I think they were really impressed.. I'll be charging them with digging deep and trying to break things.. one of them will be at Hackfest so maybe we can get him to give this a deep dive whilst there. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #13 from Jake Bateman <jake.bateman@openfifth.co.uk> --- This is a really positive change from an operations perspective, nice one Tomás. There's complexity in the details, especially the future of the SysV limb, and the potential effects on packaging and KTD. I am this year's Hackfest delegate from the O5 infrastructure team - I'd love to talk about it over coffee/beer :) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #16 from Jake Deery <jake.deery@openfifth.co.uk> --- Created attachment 196052 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=196052&action=edit Bug 40901: (testing follow-up) Remove / rename koha-common to koha-core This patch removes or renames some of the debian files from koha-common to koha-core, to avoid file clash during install -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- URL| |https://gitlab.com/koha-com | |munity/koha-misc4dev/-/merg | |e_requests/92 --- Comment #18 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Just noting that in order to launch KTD with this patches applied on your clone, you would need [this](https://gitlab.com/koha-community/koha-misc4dev/-/merge_requests/92) in KTD applied. It's a safe thing to apply to I count on having it soon. Can be dealt with by setting this in your KTD's `.env` file: ``` DEBUG_GIT_REPO_MISC4DEV=yes DEBUG_GIT_REPO_MISC4DEV_URL=https://gitlab.com/koha-community/koha-misc4dev.git DEBUG_GIT_REPO_MISC4DEV_BRANCH=issue_105 ``` -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #20 from Mason James <mtj@kohaaloha.com> --- i've built some packages with these patched added (from 98b8b535f50) and added the to the koha-staging/exp repo # cat /etc/apt/sources.list.d/koha.list deb [signed-by=/etc/apt/keyrings/koha.asc] http://debian.koha-community.org/koha-staging exp main # apt install koha-common The following additional packages will be installed: koha-core koha-l10n koha-systemd Suggested packages: letsencrypt The following NEW packages will be installed: koha-common koha-core koha-l10n koha-systemd 0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded. Need to get 57.3 MB of archives. After this operation, 787 MB of additional disk space will be used. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #22 from Jake Deery <jake.deery@openfifth.co.uk> --- Created attachment 196110 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=196110&action=edit Bug 40901: (testing follow-up) Added enable / disable to more helpers This patch adds more enable / disable flags to the koha daemon helper scripts. This is essential as systemd does not enable units by default, especially when using instantiation variables. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #23 from Jake Deery <jake.deery@openfifth.co.uk> --- Created attachment 196116 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=196116&action=edit Bug 40901: (testing follow-up) Added enable / disable to more helpers This patch adds more enable / disable flags to the koha daemon helper scripts. This is essential as systemd does not enable units by default, especially when using instantiation variables. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #25 from Jake Bateman <jake.bateman@openfifth.co.uk> --- Created attachment 196148 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=196148&action=edit Bug 40901: Add systemd try-restart to koha-systemd postinst In order for apt upgrades to trigger a service restart for all koha-systemd subservices and all instances, this adds a wildcard try-restart for those in addition to the daemon-reload. Only services which are already present and running will be restarted at postinst time -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #27 from Mason James <mtj@kohaaloha.com> ---

hi Jake, i spotted a syntax error on this patch

check out shellcheck tool (if you haven't seen it before) # apt install shellcheck # shellcheck debian/koha-systemd.postinst In debian/koha-systemd.postinst line 4: case "$1" in ^-- SC1009: The mentioned syntax error was in this case expression. In debian/koha-systemd.postinst line 11: if [ -n "$2" ]; then ^-- SC1073: Couldn't parse this case item. Fix to allow more checks. ^-- SC1072: Expected ) to open a new case item. Fix any mentioned problems and try again. ^-- SC1085: Did you forget to move the ;; after extending this case item? For more information: https://www.shellcheck.net/wiki/SC1085 -- Did you forget to move the ;; aft... https://www.shellcheck.net/wiki/SC1072 -- Expected ) to open a new case ite... https://www.shellcheck.net/wiki/SC1073 -- Couldn't parse this case item. Fi... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 cschrode <cschrode@akronlibrary.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cschrode@akronlibrary.org -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #195098|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #196051|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #196053|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #196116|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #196148|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #29 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 198244 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198244&action=edit Bug 40901: Add koha-sysv package for SysV init support This patch introduces the koha-sysv package, which provides traditional SysV init.d scripts for managing Koha instances. All services (Plack, Zebra, SIP, Z3950, workers, indexers) are managed through a single init script using the daemon wrapper. The package includes: - /etc/init.d/koha-common (SysV init script) - /lib/systemd/system/koha-common.service (systemd wrapper for init.d) This package is intended for legacy systems, containers without systemd (like KTD), or environments where SysV init is preferred. The package provides 'koha-init' and conflicts with koha-systemd to ensure only one init system is active at a time. Test plan: 1. Build and install the package 2. Start services with /etc/init.d/koha-common start 3. Verify all services start correctly 4. Check that koha-systemd cannot be installed alongside Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #31 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 198246 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198246&action=edit Bug 40901: Add support for 'systemctl' in `koha-*` scripts Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #33 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 198248 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198248&action=edit Bug 40901: (testing follow-up) Moved legacy code to koha-sysv.postinst This patch moves the update-rc.d lines from koha-core to koha-sysv, so it only fires if systemd is not in use Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #35 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 198250 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198250&action=edit Bug 40901: (testing follow-up) Added enable / disable to more helpers This patch adds more enable / disable flags to the koha daemon helper scripts. This is essential as systemd does not enable units by default, especially when using instantiation variables. Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #37 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 198252 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198252&action=edit Bug 40901: Add systemd try-restart to koha-systemd postinst In order for apt upgrades to trigger a service restart for all koha-systemd subservices and all instances, this adds a wildcard try-restart for those in addition to the daemon-reload. Only services which are already present and running will be restarted at postinst time Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #39 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 198254 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198254&action=edit Bug 40901: (follow-up) Fix incomplete systemd wiring in helpers - koha-z3950-responder: add systemd enable/disable to enable_z3950/disable_z3950 - koha-disable: add plack and es-indexer stop/disable - koha-remove: add worker --disable after --stop - koha-systemd-ctl: fix service name worker-long to worker-long_tasks Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #40 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 198265 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198265&action=edit Bug 40901: (follow-up) Extract koha-create-dirs into a oneshot unit Replace per-service ExecStartPre=+koha-create-dirs (running as root on every start of every service) with a single oneshot unit per instance. Services declare Requires= on it, and the oneshot's Before= ensures directories exist before any service starts. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #41 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 198305 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198305&action=edit Bug 40901: (follow-up) Regenerate debian/control from control.in Run debian/update-control to include koha-systemd and koha-sysv package stanzas and refresh Perl autodependencies. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #43 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 198307 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198307&action=edit Bug 40901: (follow-up) Enable systemd units for existing instances on install On first install of koha-systemd (migration from SysV), enable units for all existing Koha instances. Core services (plack, zebra, workers) are always enabled. SIP, Z39.50, and indexers are enabled conditionally based on instance configuration. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #46 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- (In reply to Phillip Berg from comment #45)

I'm not sure there's a way for me to completely convey how important fixing this bug is. We're faced with consistent stability issues and this bug is being pointed to as one of the root causes.

Hi, Phillip. This bug report is about changing how the different submodules and processes that are required for Koha as a whole are handled individually in a kosher way, if I may. This means sysadmins have full control on establishing limits for processes that go rogue, defining restart policies for individual processes that are otherwise not accountable. The case is that those individual processes policies could certainly prevent outages to last long, themselves, while providing valuable feedback for debugging and long term solutions. It will certainly highlight possible design flaws in Koha that have been hidden by a monolithic legacy script that took care of launching ‘all the things’. Most of these issues have popped up now that some seriously big libraries have stressed Koha’s capabilities to its limits. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #48 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 198328 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198328&action=edit Bug 40901: (follow-up) Read service config from koha-conf.xml in systemd units The systemd units were missing most of the parameters that the SysV init functions read from koha-conf.xml. This caused wrong worker counts, missing log files, ignored batch sizes, and other config being silently dropped. This patch adds koha-generate-env, a helper script that reads koha-conf.xml and writes an EnvironmentFile for each service. Each unit runs it as ExecStartPre and then uses the variables. Changes per unit: - koha-plack@: workers, max-requests, access/error logs, environment (development/deployment), per-instance psgi, -M FindBin - koha-zebra@: loglevels, max_record_size, timestamp format - koha-es-indexer@: batch_size - koha-indexer@: INDEXER_TIMEOUT from /etc/default/koha-common - koha-z3950@: config dir fallback to global, additional options -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #50 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Hi all. Our systems team reported some misses in the original implementation. Those are covered now, and I also revisited the upgrade path. A backwards compatible `koha-common.service` wrapper is added for a lean transition for non technical users (i.e. they retain the previous `systemctl stop/start/restart koha-common` command). I'm not sure it is 100% needed, but I added in case it can ease things for QA also. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #52 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 198554 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198554&action=edit Bug 40901: (follow-up) Fail loudly when koha-sip is enabled without flag file The koha-sip@.service unit gates startup on /var/lib/koha/<inst>/sip.enabled, the per-instance "I want SIP on" flag file. `koha-sip --enable` creates that flag in the same call that runs `systemctl enable`; running a bare `systemctl enable koha-sip@<inst>.service` only flips the systemd half, so the unit ends up enabled-but-refusing-to-start. With ConditionPathExists the failure mode is silent: the unit ends up "enabled" at the systemd level but Active: inactive (dead) with "Condition: start condition failed" and the journal logs aren't a big help figuring out the problem. This patch replaces the silent skip with a loud failure with a helpful message: ERROR: SIP not enabled for instance <inst>. Use: koha-sip --enable <inst> SIPconfig.xml is gated by AssertPathExists if it's missing the unit fails. Either failure path lands the unit in failed state, surfaced by 'systemctl --failed' Test plan: 1) Apply this patch and reload systemd: systemctl daemon-reload 2) Pick a test instance whose SIP is not enabled (no /var/lib/koha/<inst>/sip.enabled file). 3) Reproduce the trap with bare systemctl: systemctl enable koha-sip@<inst>.service systemctl start koha-sip@<inst>.service 4) Note that 'start' exits non-zero and: systemctl status koha-sip@<inst>.service reports Active: failed (Result: exit-code), and the journal contains: "ERROR: SIP not enabled for instance <inst>. Use: koha-sip --enable <inst>"! 5) Recover via the helper: systemctl disable koha-sip@<inst>.service systemctl reset-failed koha-sip@<inst>.service koha-sip --enable <inst> koha-sip --start <inst> 6) Confirm the unit is now active (running), the flag file /var/lib/koha/<inst>/sip.enabled exists, and 'koha-list --enabled --sip' lists the instance! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #53 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 198674 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198674&action=edit Bug 40901: (QA follow-up) Use --config-dir for z3950_responder.pl Documentation is wrong, -c doesn't work for z3950_responder.pl -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198243|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198245|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198247|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198249|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198251|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198253|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198265|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198306|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198327|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198366|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198674|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #55 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199074 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199074&action=edit Bug 40901: Add koha-sysv package for SysV init support This patch introduces the koha-sysv package, which provides traditional SysV init.d scripts for managing Koha instances. All services (Plack, Zebra, SIP, Z3950, workers, indexers) are managed through a single init script using the daemon wrapper. The package includes: - /etc/init.d/koha-common (SysV init script) - /lib/systemd/system/koha-common.service (systemd wrapper for init.d) This package is intended for legacy systems, containers without systemd (like KTD), or environments where SysV init is preferred. The package provides 'koha-init' and conflicts with koha-systemd to ensure only one init system is active at a time. Test plan: 1. Build and install the package 2. Start services with /etc/init.d/koha-common start 3. Verify all services start correctly 4. Check that koha-systemd cannot be installed alongside Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #57 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199076 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199076&action=edit Bug 40901: Add support for 'systemctl' in `koha-*` scripts Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #59 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199078 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199078&action=edit Bug 40901: (testing follow-up) Moved legacy code to koha-sysv.postinst This patch moves the update-rc.d lines from koha-core to koha-sysv, so it only fires if systemd is not in use Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #61 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199080 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199080&action=edit Bug 40901: (testing follow-up) Added enable / disable to more helpers This patch adds more enable / disable flags to the koha daemon helper scripts. This is essential as systemd does not enable units by default, especially when using instantiation variables. Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #63 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199082 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199082&action=edit Bug 40901: Add systemd try-restart to koha-systemd postinst In order for apt upgrades to trigger a service restart for all koha-systemd subservices and all instances, this adds a wildcard try-restart for those in addition to the daemon-reload. Only services which are already present and running will be restarted at postinst time Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #65 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199084 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199084&action=edit Bug 40901: (follow-up) Fix incomplete systemd wiring in helpers - koha-z3950-responder: add systemd enable/disable to enable_z3950/disable_z3950 - koha-disable: add plack and es-indexer stop/disable - koha-remove: add worker --disable after --stop - koha-systemd-ctl: fix service name worker-long to worker-long_tasks Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #67 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199086 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199086&action=edit Bug 40901: (follow-up) Regenerate debian/control from control.in Run debian/update-control to include koha-systemd and koha-sysv package stanzas and refresh Perl autodependencies. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #69 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199088 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199088&action=edit Bug 40901: (follow-up) Enable systemd units for existing instances on install On first install of koha-systemd (migration from SysV), enable units for all existing Koha instances. Core services (plack, zebra, workers) are always enabled. SIP, Z39.50, and indexers are enabled conditionally based on instance configuration. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #71 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199090 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199090&action=edit Bug 40901: (follow-up) Read service config from koha-conf.xml in systemd units The systemd units were missing most of the parameters that the SysV init functions read from koha-conf.xml. This caused wrong worker counts, missing log files, ignored batch sizes, and other config being silently dropped. This patch adds koha-generate-env, a helper script that reads koha-conf.xml and writes an EnvironmentFile for each service. Each unit runs it as ExecStartPre and then uses the variables. Changes per unit: - koha-plack@: workers, max-requests, access/error logs, environment (development/deployment), per-instance psgi, -M FindBin - koha-zebra@: loglevels, max_record_size, timestamp format - koha-es-indexer@: batch_size - koha-indexer@: INDEXER_TIMEOUT from /etc/default/koha-common - koha-z3950@: config dir fallback to global, additional options Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #73 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199092 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199092&action=edit Bug 40901: (follow-up) Fail loudly when koha-sip is enabled without flag file The koha-sip@.service unit gates startup on /var/lib/koha/<inst>/sip.enabled, the per-instance "I want SIP on" flag file. `koha-sip --enable` creates that flag in the same call that runs `systemctl enable`; running a bare `systemctl enable koha-sip@<inst>.service` only flips the systemd half, so the unit ends up enabled-but-refusing-to-start. With ConditionPathExists the failure mode is silent and the unit ends up "enabled" at the systemd level but "Active: inactive" (dead) with "Condition: start condition failed" and the journal logs aren't a big help figuring out the problem. This patch replaces the silent skip with a loud failure with a helpful message: ERROR: SIP not enabled for instance <inst>. Use: koha-sip --enable <inst> SIPconfig.xml is gated by AssertPathExists if it's missing the unit fails. Either failure path lands the unit in failed state, surfaced by 'systemctl --failed' Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #75 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199094 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199094&action=edit Bug 40901: (follow-up) sip.enabled is an anti-pattern for systemd Remove the use of sip.enabled for systemd, keep for sysv No other service using a .enabled file is controlled via systemd or sysv Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #77 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199096 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199096&action=edit Bug 40901: (follow-up) Allow seamless swap between koha-systemd and koha-sysv koha-systemd and koha-sysv have always declared mutual Conflicts so you can't install both, but they don't declare Replaces which would make it so installing one autmatically removes the other Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #79 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199098 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199098&action=edit Bug 40901: (follow-up) koha-remove leaves koha@<inst>.target enabled Disable koha@<inst>.target and koha-create-dirs@<inst>.service so their koha.target.wants/ symlinks don't outlive the instance. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #81 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199100 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199100&action=edit Bug 40901: (follow-up) Cap restart attempts on per-component units Add StartLimitIntervalSec=60 add StartLimitBurst=3 so a broken-at- startup daemon stops looping at after three attempts. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #83 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199102 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199102&action=edit Bug 40901: (follow-up) Tighten RestartSec on SIP, Z3950, es-indexer 30 seconds between restart attempts is a very long time especially for AIP. With StartLimitIntervalSec at 60, and RestartSec at 5 we get 6 retries and what is hopefully a less visible downtime window for the tcp services. For es-indexer it's not so big a deal but could be an issue for cataloguers. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #85 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 199104 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199104&action=edit Bug 40901: (follow-up) Systemd hardening I think the following systemd security measures are safe for Koha: ProtectSystem=full Sets /usr & /boot to read-only ProtectHome=yes Hides /home, /root and /run/user NOTE: The hiding of /run/user through me off, but google says it's only for interactive logins only. AFAIK the only thing that does that is koha-shell which isn't a systemd thing so it is unaffected PrivateTmp=yes Private /tmp & /var/tmp NOTE: AFACT nothing in Koha shares files via /tmp which feels like it would be a bad thing to do anyway NoNewPrivileges=yes setuid binaries can't re-elevate NOTE: This feels like the most important one! Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |42643 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42643 [Bug 42643] [OMNIBUS] Assorted performance and stability work -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199073|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199075|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199077|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199079|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199081|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199083|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199085|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199087|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199089|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199091|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199093|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199095|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199097|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199099|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199101|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #199103|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #86 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199424 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199424&action=edit Bug 40901: Add koha-systemd package with native systemd units This patch introduces the koha-systemd package, which provides native systemd service units for managing Koha instances. Each service type (Plack, Zebra, SIP, Z3950, workers, indexers) is managed as a separate systemd unit using instance templates. The package includes: - 8 service templates (plack, zebra, sip, z3950, worker, worker-long, indexer, es-indexer) - 2 target units (koha@.target for per-instance, koha.target for global) - Helper script (koha-systemd-ctl) for managing services All service units include: - KOHA_HOME environment variable pointing to /usr/share/koha - WorkingDirectory set to /var/lib/koha/%i - Proper user/group permissions (%i-koha) This allows granular control over individual services per instance and leverages systemd features like dependency management, resource limits, and journald logging. Test plan: 1. Build and install the package 2. Enable services for an instance 3. Verify all services start correctly 4. Check journalctl logs show no errors Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #88 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199426 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199426&action=edit Bug 40901: Restructure packages - koha-common becomes metapackage This patch restructures the Koha packaging to support both systemd and SysV init systems through a clean separation of concerns. Package hierarchy: - koha-core: Core application files (no init system) - koha-systemd: Native systemd units (Provides: koha-init) - koha-sysv: SysV init scripts (Provides: koha-init, Conflicts: koha-systemd) - koha-common: Metapackage (Depends: koha-core, koha-systemd | koha-sysv) Changes: - koha-core: Removed 'daemon' dependency, updated description to clarify it provides core files without init system - koha-systemd: Now depends on koha-core instead of koha-common, provides koha-init virtual package - koha-sysv: Now depends on koha-core instead of koha-common, provides koha-init virtual package - koha-common: Converted to metapackage that depends on koha-core and prefers koha-systemd on systemd-based systems - Removed koha-common.install (files now installed by koha-core) - Simplified koha-common.postinst (metapackage has minimal logic) Benefits: - Clean separation between core application and init system - Users can explicitly choose their init system - KTD can use koha-sysv without systemd - Modern systems get native systemd by default - Conflicts prevent both init systems being active simultaneously Test plan: 1. Build packages 2. Install koha-common on a systemd system, verify koha-systemd is pulled in 3. Try installing both koha-systemd and koha-sysv, verify conflict 4. Install koha-core + koha-sysv explicitly, verify it works Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #90 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199428 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199428&action=edit Bug 40901: (testing follow-up) Remove / rename koha-common to koha-core This patch removes or renames some of the debian files from koha-common to koha-core, to avoid file clash during install Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #92 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199430 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199430&action=edit Bug 40901: (testing follow-up) Move koha-worker-long@ unit to koha-worker-long_tasks@ We found a bug when running koha-create. The systemd unit name didn't match expected, so we've moved the unit to match the koha functions computed value. Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #94 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199432 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199432&action=edit Bug 40901: (testing follow-up) Add koha-create-dirs to units This patch adds an elevated koha-create-dirs to the systemd units, so that when the units (re)start, the directories are definitely present. We found that PID files were missing and couldn't be created without this patch. Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #96 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199434 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199434&action=edit Bug 40901: (follow-up) Fix critical bugs in systemd integration - koha-sip: disable_sip() was calling 'enable' instead of 'disable' - koha-indexer: enable/disable dispatch was calling start/stop functions - koha-systemd.postinst: upgrade try-restart was outside the case block Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #98 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199436 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199436&action=edit Bug 40901: (follow-up) Extract koha-create-dirs into a oneshot unit Replace per-service ExecStartPre=+koha-create-dirs (running as root on every start of every service) with a single oneshot unit per instance. Services declare Requires= on it, and the oneshot's Before= ensures directories exist before any service starts. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #100 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199438 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199438&action=edit Bug 40901: (follow-up) Add Replaces/Breaks for smooth upgrade from koha-common koha-core takes over files previously owned by koha-common (e.g. /etc/default/koha-common). Without Replaces/Breaks, dpkg refuses to overwrite them during upgrade. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #102 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199440 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199440&action=edit Bug 40901: (follow-up) Fix parallel execution of SysV and systemd services When upgrading from the old koha-common (pre-split) to the new koha-core + koha-systemd packages, the SysV init script /etc/init.d/koha-common is left behind as an orphan. The systemd-sysv-generator auto-creates koha-common.service from it, which runs in parallel with the native systemd units, causing double-starts of all Koha services. Changes: - koha-systemd.postinst now stops, disables, and removes the orphaned SysV init script when koha-sysv is not installed - koha-sysv gets Replaces/Breaks on koha-common (<< 25.11) so dpkg can cleanly take over the init script file Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #104 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199442 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199442&action=edit Bug 40901: (follow-up) Preserve 'service koha-common' UX with systemd Adds PartOf=koha.target to koha@.target so that stopping or restarting the global target propagates to all instance targets and their services. Ships a koha-common.service oneshot unit that delegates to koha.target, so 'systemctl restart koha-common' works as the drop-in replacement for the old 'service koha-common restart'. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #106 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199444 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199444&action=edit Bug 40901: (follow-up) Use --config-dir for z3950_responder.pl Documentation is wrong, -c doesn't work for z3950_responder.pl Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #108 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199446 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199446&action=edit Bug 40901: (follow-up) koha-create enables koha@<inst>.target on systemd Without this, freshly-created instances on a koha-systemd host do not auto-start at next boot. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #110 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199448 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199448&action=edit Bug 40901: (follow-up) Clean up stale Zebra Unix sockets on startup zebrasrv removes its Unix sockets on clean exit but leaves them behind on SIGKILL, OOM, or sysv to systemd swap, causing 'Address already in use' on the next start. ExecStartPre rm makes the unit auto-fix itself. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #112 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199450 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199450&action=edit Bug 40901: (follow-up) Re-discover ES indexers on koha-common start The SysV init script runs `koha-list --enabled --elasticsearch` at every start. ExecStartPre on koha-common.service restores that. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #114 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199452 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199452&action=edit Bug 40901: (follow-up) Order per-component units after koha-create-dirs Pair each Requires=koha-create-dirs@%i.service with a matching After=, and drop the now redundant Before= list from koha-create-dirs@.service. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #116 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199454 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199454&action=edit Bug 40901: (follow-up) Drop redundant StopWhenUnneeded on koha@.target PartOf=koha.target already cascades stop to the per-instance target so StopWhenUnneeded is not needed and adds more complication that could trip us up later. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #118 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199456 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199456&action=edit Bug 40901: (QA follow-up) Fix stale /etc/default/koha-common reference in koha-generate-env koha-generate-env sourced /etc/default/koha-common to read INDEXER_TIMEOUT, but the defaults file was renamed to /etc/default/koha-core in this patchset. On fresh installations no /etc/default/koha-common exists, so INDEXER_TIMEOUT was always silently fixed at the built-in default of 5, ignoring any admin customisation. Read /etc/default/koha-core first; fall back to /etc/default/koha-common so that systems upgrading from the pre-split packaging still pick up their existing settings. Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #120 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199458 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199458&action=edit Bug 40901: (QA follow-up) Fix koha-sysv.postinst configure guard and double-enable Two problems in the original koha-sysv.postinst: 1. #DEBHELPER# was placed at the top (line 5) before any case statement, so whatever dh_installsystemd expands it to ran for ALL postinst invocations (abort-upgrade, abort-remove, triggered…), not just configure. 2. The custom block manually called `deb-systemd-helper enable koha-common.service` on systemd hosts. Because koha-sysv also ships koha-sysv.service (which dh_installsystemd enables via #DEBHELPER#), both koha-sysv.service and the sysv-generator-created koha-common.service could end up enabled, resulting in a double-start of the init script at boot. Fix both by wrapping the custom code in `case "$1" in configure)`, moving manual enable: dh_installsystemd already enables and starts koha-sysv.service correctly. Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #122 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199460 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199460&action=edit Bug 40901: (QA follow-up) Drop redundant ExecStart systemctl call in koha-common.service koha-common.service already declares BindsTo=koha.target and After=koha.target. BindsTo= implies Requires=, so systemd will start koha.target automatically when this compat unit starts and will wait for it to be active before proceeding. The explicit ExecStart=/bin/systemctl start koha.target was therefore redundant, and calling systemctl from within a service body is an anti-pattern (it makes an extra D-Bus round-trip back to PID 1 and can deadlock in edge cases during early boot). Replace with /bin/true so the oneshot succeeds immediately after the ExecStartPre ES-indexer discovery step. ExecStop is kept as-is because there is no declarative equivalent for "stopping this unit should stop koha.target" — BindsTo= only propagates stops in the other direction. Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #124 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199462 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199462&action=edit Bug 40901: (QA follow-up) Fix koha-disable --disable idempotency under set -e koha-plack, koha-zebra, koha-indexer and koha-es-indexer all return exit code 1 when asked to disable a service that is already disabled. koha-disable has set -e, so this propagated as a fatal error when koha-create called koha-disable on a freshly-created instance (where none of the per-service configs are enabled yet). Apply the same || true guard that koha-worker already had to the four unconditional --disable calls. The desired postcondition (service disabled) is achieved either way; the exit code from the sub-script is not meaningful here. Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #126 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Still reviewing here but wanted to put up what I've worked through so far. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Failed QA --- Comment #128 from David Cook <dcook@prosentient.com.au> --- Taking a look... Just a little change eh: 57 files changed, 2099 insertions(+), 1872 deletions(-) We'll want to be careful with this bit I reckon in debian/control: 495 Package: koha-sysv 496 Architecture: all 497 Depends: ${misc:Depends}, daemon, koha-core 498 Provides: koha-init 499 Conflicts: koha-systemd 500 Replaces: koha-common (<< 25.11), koha-systemd 501 Breaks: koha-common (<< 25.11) I reckon this will break koha-full, since it won't have any init anymore. Should probably update koha-full to depend on koha-common instead of koha-full and update deps accordingly. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #130 from David Cook <dcook@prosentient.com.au> --- Has anyone tested these systemd changes yet? I'm not sure about this debian/koha-systemd.postinst. It looks a bit dodgy... -- I was wondering why koha-core doesn't have a dependency on "koha-systemd | koha-sysv", but actually that should be right, because koha-core would be the one used in containers. -- -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #132 from David Cook <dcook@prosentient.com.au> --- I'm not 100% sold on the SystemD stuff yet though. Looking at debian/control, I see this in Depends: koha-systemd | koha-sysv That means that by default Koha will try to install koha-systemd, but I think for a while we're going to want to default to koha-sysv. -- Also, what's going on in debian/systemd/koha-common.service? What's this about? ExecStartPre=+/bin/sh -c 'for i in $(koha-list --enabled --elasticsearch 2>/dev/null); do systemctl enable --now "koha-es-indexer@$i.service" 2>/dev/null || true; done' -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 --- Comment #133 from Jake Deery <jake.deery@openfifth.co.uk> ---

https://lists.debian.org/debian-user/2023/12/msg01214.html https://linuxjust4u.com/linux/systemd-260-sysv-removed-linux-upgrade-guide/

I'm normally all for a gentle introduction of change, but with SysV having (very) recently gone the way of the dodo upstream, I think pushing new installs to default to SystemD is an important decision we need to make _now_, and fix up over the cycle period ahead. To facilitate this, I've gotten together a server here at Open Fifth with _make-koha-debs_ and _nuke-all-of-koha_ scripts on the system's PATH. We'll be focused on this bug over the next few weeks, so we can make it happen. What is to happen to upgraded systems, I am more open toward. I do think, though, that in Docker, we really should start contemplating a switch to either s6 or tini as the tooling of choice for service initialisation and management. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=40901 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |43035 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43035 [Bug 43035] [OMNIBUS] System resource management and tracking -- You are receiving this mail because: You are watching all bug changes.