The many failings of background_jobs_worker.pl
Howdy! Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering. The first one was Can't call method "process" on an undefined value where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one. Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup. Then came the malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...") at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it... After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up. At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that? Regards, PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...). -- Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com>
I've also been bitten by the malformed JSON string error. On Tue, Dec 20, 2022 at 12:13 PM Philippe Blouin < philippe.blouin@inlibro.com> wrote:
Howdy!
Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering.
The first one was
Can't call method "process" on an undefined value
where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one.
Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup.
Then came the
malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...")
at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it...
After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up.
At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that?
Regards,
PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...). -- Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Michael Hafen Washington County School District Technology Department Systems Analyst
Salut Philippe, That first issue should’ve been resolved in 22.05.00 by https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. I haven’t had any problems like that since applying that patch. Are you running Koha with or without customizations? As you say, bug 30654 discusses that second issue. And I obviously have my own opinion on that one 😉. That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well I believe: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351 -- The only issue I’ve had with the background jobs has been the one covered by Bug 30172. Otherwise, it’s been all fine for me, although I use Zebra rather than Elasticsearch. I think part of the reason I haven’t had issues is that I haven’t had many people using the background jobs either though. I’m actually planning on writing a background job system based on RabbitMQ for a different non-Koha system. The main difference is that I’ll reject or fail tasks where messages aren’t sent to RabbitMQ. I think that’ll make my system a bit more robust than Koha’s. The problem with the background jobs at the moment is that we haven’t fully committed to RabbitMQ. We’re trying to do this weird hybrid with the database fallback which is not the right direction in my mind. We should do one or the other but not try to do both. But that’s just my 2 cents. David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Koha-devel <koha-devel-bounces@lists.koha-community.org> On Behalf Of Philippe Blouin Sent: Wednesday, 21 December 2022 6:13 AM To: koha-devel@lists.koha-community.org Subject: [Koha-devel] The many failings of background_jobs_worker.pl Howdy! Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering. The first one was Can't call method "process" on an undefined value where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one. Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup. Then came the malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...") at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it... After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up. At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that? Regards, PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...). -- Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 <mailto:philippe.blouin@inLibro.com> philippe.blouin@inLibro.com inLibro | pour esprit libre | <http://www.inLibro.com> www.inLibro.com
Good evening, David, Thanks for the response. Yours and David's and Michael's. I feel less alone... I validated, and yes all the patches you refer are in our pile. And until the problems arose, there were no customizations around that code. So yeah, even at 22.05.06, I get the JSON error and the race condition (we use ES). And the _abandonned_ children. So I surmise, or dare I say postulate, that those issues are not as resolved as some would presume. I will revert background_jobs_worker.pl to its default, and shutdown MQ everywhere, for now. :( Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 2022-12-20 17:55, David Cook wrote:
Salut Philippe,
That first issue should’ve been resolved in 22.05.00 by https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. I haven’t had any problems like that since applying that patch. Are you running Koha with or without customizations?
As you say, bug 30654 discusses that second issue. And I obviously have my own opinion on that one 😉.
That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well I believe: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351
--
The only issue I’ve had with the background jobs has been the one covered by Bug 30172. Otherwise, it’s been all fine for me, although I use Zebra rather than Elasticsearch. I think part of the reason I haven’t had issues is that I haven’t had many people using the background jobs either though.
I’m actually planning on writing a background job system based on RabbitMQ for a different non-Koha system. The main difference is that I’ll reject or fail tasks where messages aren’t sent to RabbitMQ. I think that’ll make my system a bit more robust than Koha’s.
The problem with the background jobs at the moment is that we haven’t fully committed to RabbitMQ. We’re trying to do this weird hybrid with the database fallback which is not the right direction in my mind. We should do one or the other but not try to do both.
But that’s just my 2 cents.
David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*Koha-devel <koha-devel-bounces@lists.koha-community.org> *On Behalf Of *Philippe Blouin *Sent:* Wednesday, 21 December 2022 6:13 AM *To:* koha-devel@lists.koha-community.org *Subject:* [Koha-devel] The many failings of background_jobs_worker.pl
Howdy!
Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering.
The first one was
Can't call method "process" on an undefined value
where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one.
Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup.
Then came the
malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...")
at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it...
After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up.
At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that?
Regards,
PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...).
--
Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com <mailto:philippe.blouin@inLibro.com>
inLibro| pour esprit libre |www.inLibro.com <http://www.inLibro.com>
Although I precise, Cannot connect to broker Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. So shutting down MQ has its own issues.... Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 2022-12-21 08:58, Philippe Blouin wrote:
Good evening, David,
Thanks for the response. Yours and David's and Michael's. I feel less alone...
I validated, and yes all the patches you refer are in our pile. And until the problems arose, there were no customizations around that code.
So yeah, even at 22.05.06, I get the JSON error and the race condition (we use ES). And the _abandonned_ children. So I surmise, or dare I say postulate, that those issues are not as resolved as some would presume.
I will revert background_jobs_worker.pl to its default, and shutdown MQ everywhere, for now. :(
Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 2022-12-20 17:55, David Cook wrote:
Salut Philippe,
That first issue should’ve been resolved in 22.05.00 by https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. I haven’t had any problems like that since applying that patch. Are you running Koha with or without customizations?
As you say, bug 30654 discusses that second issue. And I obviously have my own opinion on that one 😉.
That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well I believe: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351
--
The only issue I’ve had with the background jobs has been the one covered by Bug 30172. Otherwise, it’s been all fine for me, although I use Zebra rather than Elasticsearch. I think part of the reason I haven’t had issues is that I haven’t had many people using the background jobs either though.
I’m actually planning on writing a background job system based on RabbitMQ for a different non-Koha system. The main difference is that I’ll reject or fail tasks where messages aren’t sent to RabbitMQ. I think that’ll make my system a bit more robust than Koha’s.
The problem with the background jobs at the moment is that we haven’t fully committed to RabbitMQ. We’re trying to do this weird hybrid with the database fallback which is not the right direction in my mind. We should do one or the other but not try to do both.
But that’s just my 2 cents.
David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*Koha-devel <koha-devel-bounces@lists.koha-community.org> *On Behalf Of *Philippe Blouin *Sent:* Wednesday, 21 December 2022 6:13 AM *To:* koha-devel@lists.koha-community.org *Subject:* [Koha-devel] The many failings of background_jobs_worker.pl
Howdy!
Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering.
The first one was
Can't call method "process" on an undefined value
where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one.
Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup.
Then came the
malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...")
at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it...
After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up.
At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that?
Regards,
PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...).
--
Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com <mailto:philippe.blouin@inLibro.com>
inLibro| pour esprit libre |www.inLibro.com <http://www.inLibro.com>
I dont think you can shut down rabbitmq and expect things to work. rabbitmq needs to be running and this command should return 4 processes (at least it does on our systems) ``ps ax|grep 'background_jobs_worker\.pl.*--queue.*\(default\|long_tasks\)'`` 1016012 ? S 0:00 daemon --name=inlibro-koha-worker --errlog=/var/log/koha/inlibro/worker-error.log --stdout=/var/log/koha/inlibro/worker.log --output=/var/log/koha/inlibro/worker-output.log --pidfiles=/var/run/koha/inlibro/ --verbose=1 --respawn --delay=30 --user=inlibro-koha.inlibro-koha -- /usr/share/koha/bin/background_jobs_worker.pl --queue default 1016014 ? S 0:37 /usr/bin/perl /usr/share/koha/bin/background_jobs_worker.pl --queue default 1016040 ? S 0:00 daemon --name=inlibro-koha-worker-long_tasks --errlog=/var/log/koha/inlibro/worker-error.log --stdout=/var/log/koha/inlibro/worker.log --output=/var/log/koha/inlibro/worker-output.log --pidfiles=/var/run/koha/inlibro/ --verbose=1 --respawn --delay=30 --user=inlibro-koha.inlibro-koha -- /usr/share/koha/bin/background_jobs_worker.pl --queue long_tasks 1016042 ? S 0:01 /usr/bin/perl /usr/share/koha/bin/background_jobs_worker.pl --queue long_tasks `` regards david On Wed, 21 Dec 2022, at 3:04 PM, Philippe Blouin wrote:
Although I precise,
Cannot connect to broker Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27.
So shutting down MQ has its own issues....
Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com <http://www.inlibro.com/> On 2022-12-21 08:58, Philippe Blouin wrote:
Good evening, David,
Thanks for the response. Yours and David's and Michael's. I feel less alone...
I validated, and yes all the patches you refer are in our pile. And until the problems arose, there were no customizations around that code.
So yeah, even at 22.05.06, I get the JSON error and the race condition (we use ES). And the _abandonned_ children. So I surmise, or dare I say postulate, that those issues are not as resolved as some would presume.
I will revert background_jobs_worker.pl to its default, and shutdown MQ everywhere, for now. :(
Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com <http://www.inlibro.com/> On 2022-12-20 17:55, David Cook wrote:
Salut Philippe,
That first issue should’ve been resolved in 22.05.00 by https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. I haven’t had any problems like that since applying that patch. Are you running Koha with or without customizations?
As you say, bug 30654 discusses that second issue. And I obviously have my own opinion on that one 😉.
That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well I believe: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351
--
The only issue I’ve had with the background jobs has been the one covered by Bug 30172. Otherwise, it’s been all fine for me, although I use Zebra rather than Elasticsearch. I think part of the reason I haven’t had issues is that I haven’t had many people using the background jobs either though.
I’m actually planning on writing a background job system based on RabbitMQ for a different non-Koha system. The main difference is that I’ll reject or fail tasks where messages aren’t sent to RabbitMQ. I think that’ll make my system a bit more robust than Koha’s.
The problem with the background jobs at the moment is that we haven’t fully committed to RabbitMQ. We’re trying to do this weird hybrid with the database fallback which is not the right direction in my mind. We should do one or the other but not try to do both.
But that’s just my 2 cents.
David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
*From:* Koha-devel <koha-devel-bounces@lists.koha-community.org> *On Behalf Of *Philippe Blouin *Sent:* Wednesday, 21 December 2022 6:13 AM *To:* koha-devel@lists.koha-community.org *Subject:* [Koha-devel] The many failings of background_jobs_worker.pl
Howdy!
Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering.
The first one was
Can't call method "process" on an undefined value
where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one.
Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup.
Then came the
malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...")
at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it...
After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up.
At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that?
Regards,
PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...).
-- Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com <http://www.inlibro.com/>
Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
My understanding is that all tasks go to the DB right now, so shutting down MQ doesn't lose anything. Right now. But having MQ running _does_ lose me some things, since not all tasks are processed, and the process can even get stuck in limbo with the UTF8 issue. With tens of installations to monitor, some putting thousands of jobs on the queue each day, I'm really looking for a quick fix before the Holidays... Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 2022-12-21 09:15, David Schmidt wrote:
I dont think you can shut down rabbitmq and expect things to work.
rabbitmq needs to be running and this command should return 4 processes (at least it does on our systems)
`|ps ax|grep 'background_jobs_worker\.pl.*--queue.*\(default\|long_tasks\)'`| 1016012 ? S 0:00 daemon --name=inlibro-koha-worker --errlog=/var/log/koha/inlibro/worker-error.log --stdout=/var/log/koha/inlibro/worker.log --output=/var/log/koha/inlibro/worker-output.log --pidfiles=/var/run/koha/inlibro/ --verbose=1 --respawn --delay=30 --user=inlibro-koha.inlibro-koha -- /usr/share/koha/bin/background_jobs_worker.pl --queue default 1016014 ? S 0:37 /usr/bin/perl /usr/share/koha/bin/background_jobs_worker.pl --queue default 1016040 ? S 0:00 daemon --name=inlibro-koha-worker-long_tasks --errlog=/var/log/koha/inlibro/worker-error.log --stdout=/var/log/koha/inlibro/worker.log --output=/var/log/koha/inlibro/worker-output.log --pidfiles=/var/run/koha/inlibro/ --verbose=1 --respawn --delay=30 --user=inlibro-koha.inlibro-koha -- /usr/share/koha/bin/background_jobs_worker.pl --queue long_tasks 1016042 ? S 0:01 /usr/bin/perl /usr/share/koha/bin/background_jobs_worker.pl --queue long_tasks ||
regards david
On Wed, 21 Dec 2022, at 3:04 PM, Philippe Blouin wrote:
Although I precise,
Cannot connect to broker Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27.
So shutting down MQ has its own issues....
Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com <mailto:philippe.blouin@inLibro.com> inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 2022-12-21 08:58, Philippe Blouin wrote:
Good evening, David,
Thanks for the response. Yours and David's and Michael's. I feel less alone...
I validated, and yes all the patches you refer are in our pile. And until the problems arose, there were no customizations around that code.
So yeah, even at 22.05.06, I get the JSON error and the race condition (we use ES). And the _abandonned_ children. So I surmise, or dare I say postulate, that those issues are not as resolved as some would presume.
I will revert background_jobs_worker.pl to its default, and shutdown MQ everywhere, for now. :(
Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com <mailto:philippe.blouin@inLibro.com> inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 2022-12-20 17:55, David Cook wrote:
Salut Philippe,
That first issue should’ve been resolved in 22.05.00 by https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172>. I haven’t had any problems like that since applying that patch. Are you running Koha with or without customizations?
As you say, bug 30654 discusses that second issue. And I obviously have my own opinion on that one 😉.
That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well I believe: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351 <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351>
--
The only issue I’ve had with the background jobs has been the one covered by Bug 30172. Otherwise, it’s been all fine for me, although I use Zebra rather than Elasticsearch. I think part of the reason I haven’t had issues is that I haven’t had many people using the background jobs either though.
I’m actually planning on writing a background job system based on RabbitMQ for a different non-Koha system. The main difference is that I’ll reject or fail tasks where messages aren’t sent to RabbitMQ. I think that’ll make my system a bit more robust than Koha’s.
The problem with the background jobs at the moment is that we haven’t fully committed to RabbitMQ. We’re trying to do this weird hybrid with the database fallback which is not the right direction in my mind. We should do one or the other but not try to do both.
But that’s just my 2 cents.
David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*Koha-devel <koha-devel-bounces@lists.koha-community.org> <mailto:koha-devel-bounces@lists.koha-community.org> *On Behalf Of *Philippe Blouin *Sent:* Wednesday, 21 December 2022 6:13 AM *To:* koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> *Subject:* [Koha-devel] The many failings of background_jobs_worker.pl
Howdy!
Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering.
The first one was
Can't call method "process" on an undefined value
where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one.
Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup.
Then came the
malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...")
at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it...
After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up.
At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that?
Regards,
PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...).
--
Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com <mailto:philippe.blouin@inLibro.com>
inLibro| pour esprit libre |www.inLibro.com <http://www.inLibro.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website :https://www.koha-community.org/ git :https://git.koha-community.org/ bugs :https://bugs.koha-community.org/
Something to investigate. I tend to stick to the *.11 releases, so I’ll see with 22.11 if we have any message queue issues. David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Philippe Blouin <philippe.blouin@inlibro.com> Sent: Thursday, 22 December 2022 1:04 AM To: David Cook <dcook@prosentient.com.au>; koha-devel@lists.koha-community.org Subject: Re: [Koha-devel] The many failings of background_jobs_worker.pl Although I precise, Cannot connect to broker Failed to connect: Error connecting to localhost:61613: Connection refused at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. So shutting down MQ has its own issues.... Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 <mailto:philippe.blouin@inLibro.com> philippe.blouin@inLibro.com inLibro | pour esprit libre | <http://www.inLibro.com> www.inLibro.com On 2022-12-21 08:58, Philippe Blouin wrote: Good evening, David, Thanks for the response. Yours and David's and Michael's. I feel less alone... I validated, and yes all the patches you refer are in our pile. And until the problems arose, there were no customizations around that code. So yeah, even at 22.05.06, I get the JSON error and the race condition (we use ES). And the abandonned children. So I surmise, or dare I say postulate, that those issues are not as resolved as some would presume. I will revert background_jobs_worker.pl to its default, and shutdown MQ everywhere, for now. :( Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 <mailto:philippe.blouin@inLibro.com> philippe.blouin@inLibro.com inLibro | pour esprit libre | <http://www.inLibro.com> www.inLibro.com On 2022-12-20 17:55, David Cook wrote: Salut Philippe, That first issue should’ve been resolved in 22.05.00 by https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. I haven’t had any problems like that since applying that patch. Are you running Koha with or without customizations? As you say, bug 30654 discusses that second issue. And I obviously have my own opinion on that one 😉. That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well I believe: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351 -- The only issue I’ve had with the background jobs has been the one covered by Bug 30172. Otherwise, it’s been all fine for me, although I use Zebra rather than Elasticsearch. I think part of the reason I haven’t had issues is that I haven’t had many people using the background jobs either though. I’m actually planning on writing a background job system based on RabbitMQ for a different non-Koha system. The main difference is that I’ll reject or fail tasks where messages aren’t sent to RabbitMQ. I think that’ll make my system a bit more robust than Koha’s. The problem with the background jobs at the moment is that we haven’t fully committed to RabbitMQ. We’re trying to do this weird hybrid with the database fallback which is not the right direction in my mind. We should do one or the other but not try to do both. But that’s just my 2 cents. David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia Office: 02 9212 0899 Online: 02 8005 0595 From: Koha-devel <mailto:koha-devel-bounces@lists.koha-community.org> <koha-devel-bounces@lists.koha-community.org> On Behalf Of Philippe Blouin Sent: Wednesday, 21 December 2022 6:13 AM To: koha-devel@lists.koha-community.org <mailto:koha-devel@lists.koha-community.org> Subject: [Koha-devel] The many failings of background_jobs_worker.pl Howdy! Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering. The first one was Can't call method "process" on an undefined value where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one. Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup. Then came the malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...") at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it... After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up. At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that? Regards, PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...). -- Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 <mailto:philippe.blouin@inLibro.com> philippe.blouin@inLibro.com inLibro | pour esprit libre | <http://www.inLibro.com> www.inLibro.com
You need to understand that you have dangling messages on your 🐇. Old messages that have invalid JSON, and ids that were picked by the hybrid fallback mechanism so fail (which I agree it's not cool) Stop Koha, stop the worker, stop rabbit, clean rabbit, start the worker, let it exhaust already injected tasks. Stop the worker, start rabbit, start the worker, start Koha. Sounds too much, doesn't it? El mié, 21 dic 2022 10:59, Philippe Blouin <philippe.blouin@inlibro.com> escribió:
Good evening, David,
Thanks for the response. Yours and David's and Michael's. I feel less alone...
I validated, and yes all the patches you refer are in our pile. And until the problems arose, there were no customizations around that code.
So yeah, even at 22.05.06, I get the JSON error and the race condition (we use ES). And the *abandonned* children. So I surmise, or dare I say postulate, that those issues are not as resolved as some would presume.
I will revert background_jobs_worker.pl to its default, and shutdown MQ everywhere, for now. :( Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com On 2022-12-20 17:55, David Cook wrote:
Salut Philippe,
That first issue should’ve been resolved in 22.05.00 by https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. I haven’t had any problems like that since applying that patch. Are you running Koha with or without customizations?
As you say, bug 30654 discusses that second issue. And I obviously have my own opinion on that one 😉.
That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well I believe: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351
--
The only issue I’ve had with the background jobs has been the one covered by Bug 30172. Otherwise, it’s been all fine for me, although I use Zebra rather than Elasticsearch. I think part of the reason I haven’t had issues is that I haven’t had many people using the background jobs either though.
I’m actually planning on writing a background job system based on RabbitMQ for a different non-Koha system. The main difference is that I’ll reject or fail tasks where messages aren’t sent to RabbitMQ. I think that’ll make my system a bit more robust than Koha’s.
The problem with the background jobs at the moment is that we haven’t fully committed to RabbitMQ. We’re trying to do this weird hybrid with the database fallback which is not the right direction in my mind. We should do one or the other but not try to do both.
But that’s just my 2 cents.
David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:* Koha-devel <koha-devel-bounces@lists.koha-community.org> <koha-devel-bounces@lists.koha-community.org> *On Behalf Of *Philippe Blouin *Sent:* Wednesday, 21 December 2022 6:13 AM *To:* koha-devel@lists.koha-community.org *Subject:* [Koha-devel] The many failings of background_jobs_worker.pl
Howdy!
Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering.
The first one was
Can't call method "process" on an undefined value
where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one.
Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup.
Then came the
malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...")
at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it...
After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up.
At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that?
Regards,
PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...).
--
Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Could you define "the hybrid fallback" ? I'm not sure where those old messages would come from, on a clean install. And why would they occur randomly every few hundred or thousand messages, or not at all. This would explain the JSON issue (msg put there in 21.05, collected in 22.05 ?). But I can't see how this scenario would occur on a freshly installed MQ. Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 2022-12-21 14:52, Tomas Cohen Arazi wrote:
You need to understand that you have dangling messages on your 🐇.
Old messages that have invalid JSON, and ids that were picked by the hybrid fallback mechanism so fail (which I agree it's not cool)
Stop Koha, stop the worker, stop rabbit, clean rabbit, start the worker, let it exhaust already injected tasks. Stop the worker, start rabbit, start the worker, start Koha.
Sounds too much, doesn't it?
El mié, 21 dic 2022 10:59, Philippe Blouin <philippe.blouin@inlibro.com> escribió:
Good evening, David,
Thanks for the response. Yours and David's and Michael's. I feel less alone...
I validated, and yes all the patches you refer are in our pile. And until the problems arose, there were no customizations around that code.
So yeah, even at 22.05.06, I get the JSON error and the race condition (we use ES). And the _abandonned_ children. So I surmise, or dare I say postulate, that those issues are not as resolved as some would presume.
I will revert background_jobs_worker.pl <http://background_jobs_worker.pl> to its default, and shutdown MQ everywhere, for now. :(
Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 2022-12-20 17:55, David Cook wrote:
Salut Philippe,
That first issue should’ve been resolved in 22.05.00 by https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172. I haven’t had any problems like that since applying that patch. Are you running Koha with or without customizations?
As you say, bug 30654 discusses that second issue. And I obviously have my own opinion on that one 😉.
That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as well I believe: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351
--
The only issue I’ve had with the background jobs has been the one covered by Bug 30172. Otherwise, it’s been all fine for me, although I use Zebra rather than Elasticsearch. I think part of the reason I haven’t had issues is that I haven’t had many people using the background jobs either though.
I’m actually planning on writing a background job system based on RabbitMQ for a different non-Koha system. The main difference is that I’ll reject or fail tasks where messages aren’t sent to RabbitMQ. I think that’ll make my system a bit more robust than Koha’s.
The problem with the background jobs at the moment is that we haven’t fully committed to RabbitMQ. We’re trying to do this weird hybrid with the database fallback which is not the right direction in my mind. We should do one or the other but not try to do both.
But that’s just my 2 cents.
David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
*From:*Koha-devel <koha-devel-bounces@lists.koha-community.org> <mailto:koha-devel-bounces@lists.koha-community.org> *On Behalf Of *Philippe Blouin *Sent:* Wednesday, 21 December 2022 6:13 AM *To:* koha-devel@lists.koha-community.org *Subject:* [Koha-devel] The many failings of background_jobs_worker.pl <http://background_jobs_worker.pl>
Howdy!
Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering.
The first one was
Can't call method "process" on an undefined value
where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one.
Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup.
Then came the
malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...")
at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it...
After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up.
At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that?
Regards,
PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...).
--
Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com <mailto:philippe.blouin@inLibro.com>
inLibro| pour esprit libre |www.inLibro.com <http://www.inLibro.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
jup, the JSON bug hit us too we also encountered some problems regarding rabbitmq/workers debugging wasn't straigthforward because I had a hard time to find meaningful debug output. luckily the folks on IRC were very helpful as usual. the latest bug (not sure if it is a bug as of yet) was that the indexing job for manually created biblios wasnt triggered unless zebra was running (we use elasticsearch) overall my confidence in the implementation quality of the rabbitmq/worker feature isnt overwhelming. that's not a complaint and I dont mean to step on anyones toes, I know that it's OSS and thus up to me to put in the work if im unhappy. and as always there is the possibility that I did something wrong and the software is perfectly fine. :) cheers david On Tue, 20 Dec 2022, at 8:13 PM, Philippe Blouin wrote:
Howdy!
Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering.
The first one was
Can't call method "process" on an undefined value
where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one.
Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup.
Then came the
malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...")
at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it...
After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up.
At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that?
Regards,
PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...).
--
Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com <http://www.inlibro.com/> _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
Hi, I think network performance is really important for RabbitMQ. We at Biblibre have a virtual machine in each server, to share between each virtual machine (one Koha per machine) but keep a good network performance. Looks to work well, but we are still in 21.11. Best regards, Le 20/12/2022 à 09:13, Philippe Blouin a écrit :
Howdy!
Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering.
The first one was
Can't call method "process" on an undefined value
where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one.
Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup.
Then came the
malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...")
at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it...
After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up.
At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that?
Regards,
PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...).
-- Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
-- Fridolin SOMERS <fridolin.somers@biblibre.com> Software and system maintainer 🦄 BibLibre, France
Hello Fridolin, Yes, performance is probably important for MQ, it seems to eat a disproportionate amount of resources sometime. :) But like I said, our server is a beast. In every aspect of its build, it has the best components for 2022. Let's see about the race condition: * An update is done on a biblio * An update_elastic_index job is created in the db. * Its ID is pushed onto MQ * background_jobs_worker.pl picks up the ID from MQ o it goes to the DB, and finds nothing with that ID. We get a pointer error (yeah, I come from C) o This is NOT an old forgotten floating job, since we can see the job in the database when looking manually. o The jobs stays there forever, with status 'new'. * If I add a "sleep 1", this issue _mostly_ disappear. There's no server performance that could explain this. Maybe some DB caching ? Philippe Blouin, Directeur de la technologie Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com> On 2022-12-21 13:06, Fridolin SOMERS wrote:
Hi,
I think network performance is really important for RabbitMQ. We at Biblibre have a virtual machine in each server, to share between each virtual machine (one Koha per machine) but keep a good network performance. Looks to work well, but we are still in 21.11.
Best regards,
Le 20/12/2022 à 09:13, Philippe Blouin a écrit :
Howdy!
Since moving a lot of our users to 22.05.06, we've installed the worker everywhere. But the number of issues encountered is staggering.
The first one was
Can't call method "process" on an undefined value
where the id received from MQ was not found in the DB, and the process is going straight to process_job and failing. Absolutely no idea how that occurs, seems completely counterintuitive (the ID comes from the DB after all), but here it is. Hacked the code to add a "sleep 1" to fix most of that one.
Then came the fact that stored events were not checked if the connection to MQ was successful at startup. Bug 30654 refers it. Hacked a little "$init" in there to clear that up at startup.
Then came the
malformed UTF-8 character in JSON string, at character offset 296 (before "\x{e9}serv\x{e9} au ...")
at decode_json that crashes the whole process. And for some reason, it never gets over it, gets the same problem at every restart, like the event is never "eaten" from the queue. Hacked an eval then a try-catch over it...
After coding a monitor to alert when a background_jobs has been "new" over 5 minutes in the DB, I was inundated by messages. There's alway one elasticsearch_update that escapes among the flurry, and they slowly add up.
At this point, the only viable solution is to run the workers but disable RabbitMQ everywhere. Are we really the only ones experiencing that?
Regards,
PS Our servers are well-above-average Debian 11 machines with lot of firepower (ram, cpu, i/o...).
-- Philippe Blouin, Directeur de la technologie
Tél. : (833) 465-4276, poste 230 philippe.blouin@inLibro.com
inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : https://www.koha-community.org/ git : https://git.koha-community.org/ bugs : https://bugs.koha-community.org/
participants (6)
-
David Cook -
David Schmidt -
Fridolin SOMERS -
Michael Hafen -
Philippe Blouin -
Tomas Cohen Arazi