[Koha-devel] Background jobs quirks

Thu Oct 19 06:55:21 CEST 2023

On a standard Debian package install of Koha, it looks like there are 3
persistent connections to the database: 2x Starman workers and 1x Zebra
Indexer.

Then if you use the background jobs, the background job worker for that
queue will create a persistent connection as well. 

I had to work out the MySQL/MariaDB ID through trial and error since there
wasn't enough information in the processlist in the database, but I got the
ID, so I'll check the logs in the morning to see if that's the problem. 

--

I'm losing some confidence in this theory as I'm looking at the
koha-testing-docker database instance and I'm seeing similar timeouts, but
it hasn't triggered the "killed by signal 13, respawning".

2023-10-19  3:06:09 44 [Warning] Aborted connection 44 to db: 'koha_kohadev'
user: 'koha_kohadev' host: '172.21.0.4' (Got an error reading communication
packets)

2023-10-19  3:06:09 32 [Warning] Aborted connection 32 to db: 'koha_kohadev'
user: 'koha_kohadev' host: '172.21.0.4' (Got an error reading communication
packets)

--

I'm thinking the best thing to do is just ignore SIGPIPE and let our regular
error handling take care of it, as we should then get a more nuanced error
message somewhere.

Looking at https://metacpan.org/dist/Net-Stomp/source/lib/Net/Stomp.pm and
it ignores SIGPIPE when it's sending frames. so it's probably not the
$conn->ack() doing it.

I do wonder if it's "my $job = Koha::BackgroundJobs->find($args->{job_id});"

I suppose we'll see in the morning.

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

Office: 02 9212 0899

Online: 02 8005 0595

From: Koha-devel <koha-devel-bounces at lists.koha-community.org> On Behalf Of
David Cook via Koha-devel
Sent: Thursday, October 19, 2023 2:41 PM
To: 'Koha Devel' <koha-devel at lists.koha-community.org>
Subject: [Koha-devel] Background jobs quirks

Hi all,

Have you seen something like the following in your worker-error.log?

20231018 07:44:06 instance-koha-worker-long_tasks: client (pid 3949888)
killed by signal 13, respawning

20231019 07:21:26 instance-koha-worker-long_tasks: client (pid 4082852)
killed by signal 13, respawning

I found some discussion in a thread
https://www.mail-archive.com/koha@lists.katipo.co.nz/msg30046.html

Signal 13 would be a SIGPIPE I believe. At first, I assumed it was the
connection between the background worker and RabbitMQ, but based off the
RabbitMQ logs[1] that seemed unlikely.

MariaDB is another possibility. Looking at its logs [2] and I see some timed
out connections. 

I might monitor the connections between the background workers and the
database tonight and see if it's related. It might also explain why it
happens on my MariaDB system but not my MySQL system.

Anyway, just putting out some feelers.

[1]

2023-10-18 07:44:06.492400+11:00 [info] <0.30487.131> accepting STOMP
connection <0.30487.131> (127.0.0.1:59120 -> 127.0.0.1:61613)

2023-10-18 07:44:06.500534+11:00 [info] <0.30487.131> closing STOMP
connection <0.30487.131> (127.0.0.1:59120 -> 127.0.0.1:61613)

2023-10-18 07:44:06.510426+11:00 [info] <0.20712.130> closing STOMP
connection <0.20712.130> (127.0.0.1:57508 -> 127.0.0.1:61613)

2023-10-18 07:44:08.703380+11:00 [info] <0.30518.131> accepting STOMP
connection <0.30518.131> (127.0.0.1:59882 -> 127.0.0.1:61613)

2023-10-19 07:21:26.254065+11:00 [info] <0.27344.133> accepting STOMP
connection <0.27344.133> (127.0.0.1:54116 -> 127.0.0.1:61613)

2023-10-19 07:21:26.259129+11:00 [info] <0.27344.133> closing STOMP
connection <0.27344.133> (127.0.0.1:54116 -> 127.0.0.1:61613)

2023-10-19 07:21:26.269652+11:00 [info] <0.30518.131> closing STOMP
connection <0.30518.131> (127.0.0.1:59882 -> 127.0.0.1:61613)

2023-10-19 07:21:26.660604+11:00 [info] <0.27372.133> accepting STOMP
connection <0.27372.133> (127.0.0.1:46178 -> 127.0.0.1:61613)

[2]

Oct 18 02:08:28 awesome-host mariadbd[959]: 2023-10-18  2:08:28 6308
[Warning] Aborted connection 6308 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 18 02:15:58 awesome-host mariadbd[959]: 2023-10-18  2:15:58 6304
[Warning] Aborted connection 6304 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 18 02:16:06 awesome-host mariadbd[959]: 2023-10-18  2:16:06 6303
[Warning] Aborted connection 6303 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 18 21:10:08 awesome-host mariadbd[959]: 2023-10-18 21:10:08 6380
[Warning] Aborted connection 6380 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 19 02:21:17 awesome-host mariadbd[959]: 2023-10-19  2:21:17 6431
[Warning] Aborted connection 6431 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 19 02:24:02 awesome-host mariadbd[959]: 2023-10-19  2:24:02 6432
[Warning] Aborted connection 6432 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

Office: 02 9212 0899

Online: 02 8005 0595

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.koha-community.org/pipermail/koha-devel/attachments/20231019/17a41459/attachment-0001.htm>