[Koha-devel] The many failings of background_jobs_worker.pl

Wed Dec 21 21:33:54 CET 2022

Could you define "the hybrid fallback" ?

I'm not sure where those old messages would come from, on a clean 
install.  And why would they occur randomly every few hundred or 
thousand messages, or not at all.

This would explain the JSON issue (msg put there in 21.05, collected in 
22.05 ?). But I can't see how this scenario would occur on a freshly 
installed MQ.

Philippe Blouin,
Directeur de la technologie

Tél.  : (833) 465-4276, poste 230
philippe.blouin at inLibro.com

inLibro | pour esprit libre | www.inLibro.com <http://www.inLibro.com>
On 2022-12-21 14:52, Tomas Cohen Arazi wrote:
> You need to understand that you have dangling messages on your 🐇.
>
> Old messages that have invalid JSON, and ids that were picked by the 
> hybrid fallback mechanism so fail (which I agree it's not cool)
>
> Stop Koha, stop the worker, stop rabbit, clean rabbit, start the 
> worker, let it exhaust already injected tasks. Stop the worker, start 
> rabbit, start the worker, start Koha.
>
> Sounds too much, doesn't it?
>
>
>
> El mié, 21 dic 2022 10:59, Philippe Blouin 
> <philippe.blouin at inlibro.com> escribió:
>
>     Good evening, David,
>
>     Thanks for the response.  Yours and David's and Michael's.  I feel
>     less alone...
>
>     I validated, and yes all the patches you refer are in our pile. 
>     And until the problems arose, there were no customizations around
>     that code.
>
>     So yeah, even at 22.05.06, I get the JSON error and the race
>     condition (we use ES).  And the _abandonned_ children.  So I
>     surmise, or dare I say postulate, that those issues are not as
>     resolved as some would presume.
>
>     I will revert background_jobs_worker.pl
>     <http://background_jobs_worker.pl> to its default, and shutdown MQ
>     everywhere, for now.  :(
>
>     Philippe Blouin,
>     Directeur de la technologie
>
>     Tél.  : (833) 465-4276, poste 230
>     philippe.blouin at inLibro.com
>
>     inLibro | pour esprit libre | www.inLibro.com
>     <http://www.inLibro.com>
>     On 2022-12-20 17:55, David Cook wrote:
>>
>>     Salut Philippe,
>>
>>     That first issue should’ve been resolved in 22.05.00 by
>>     https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172.
>>     I haven’t had any problems like that since applying that patch.
>>     Are you running Koha with or without customizations?
>>
>>     As you say, bug 30654 discusses that second issue. And I
>>     obviously have my own opinion on that one 😉.
>>
>>     That JSON issue should be fixed by Bug 31351 in Koha 22.05.06 as
>>     well I believe:
>>     https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31351
>>
>>     --
>>
>>     The only issue I’ve had with the background jobs has been the one
>>     covered by Bug 30172. Otherwise, it’s been all fine for me,
>>     although I use Zebra rather than Elasticsearch. I think part of
>>     the reason I haven’t had issues is that I haven’t had many people
>>     using the background jobs either though.
>>
>>     I’m actually planning on writing a background job system based on
>>     RabbitMQ for a different non-Koha system. The main difference is
>>     that I’ll reject or fail tasks where messages aren’t sent to
>>     RabbitMQ. I think that’ll make my system a bit more robust than
>>     Koha’s.
>>
>>     The problem with the background jobs at the moment is that we
>>     haven’t fully committed to RabbitMQ. We’re trying to do this
>>     weird hybrid with the database fallback which is not the right
>>     direction in my mind. We should do one or the other but not try
>>     to do both.
>>
>>     But that’s just my 2 cents.
>>
>>     David Cook
>>
>>     Senior Software Engineer
>>
>>     Prosentient Systems
>>
>>     Suite 7.03
>>
>>     6a Glen St
>>
>>     Milsons Point NSW 2061
>>
>>     Australia
>>
>>     Office: 02 9212 0899
>>
>>     Online: 02 8005 0595
>>
>>     *From:*Koha-devel <koha-devel-bounces at lists.koha-community.org>
>>     <mailto:koha-devel-bounces at lists.koha-community.org> *On Behalf
>>     Of *Philippe Blouin
>>     *Sent:* Wednesday, 21 December 2022 6:13 AM
>>     *To:* koha-devel at lists.koha-community.org
>>     *Subject:* [Koha-devel] The many failings of
>>     background_jobs_worker.pl <http://background_jobs_worker.pl>
>>
>>     Howdy!
>>
>>     Since moving a lot of our users to 22.05.06, we've installed the
>>     worker everywhere.  But the number of issues encountered is
>>     staggering.
>>
>>     The first one was
>>
>>     Can't call method "process" on an undefined value
>>
>>     where the id received from MQ was not found in the DB, and the
>>     process is going straight to process_job and failing.  Absolutely
>>     no idea how that occurs, seems completely counterintuitive (the
>>     ID comes from the DB after all), but here it is.  Hacked the code
>>     to add a "sleep 1" to fix most of that one.
>>
>>     Then came the fact that stored events were not checked if the
>>     connection to MQ was successful at startup.  Bug 30654 refers
>>     it.  Hacked a little "$init" in there to clear that up at startup.
>>
>>     Then came the
>>
>>     malformed UTF-8 character in JSON string, at character offset 296
>>     (before "\x{e9}serv\x{e9} au ...")
>>
>>     at decode_json that crashes the whole process.  And for some
>>     reason, it never gets over it, gets the same problem at every
>>     restart, like the event is never "eaten" from the queue.  Hacked
>>     an eval then a try-catch over it...
>>
>>     After coding a monitor to alert when a background_jobs has been
>>     "new" over 5 minutes in the DB, I was inundated by messages. 
>>     There's alway one elasticsearch_update that escapes among the
>>     flurry, and they slowly add up.
>>
>>     At this point, the only viable solution is to run the workers but
>>     disable RabbitMQ everywhere.  Are we really the only ones
>>     experiencing that?
>>
>>     Regards,
>>
>>     PS Our servers are well-above-average Debian 11 machines with lot
>>     of firepower (ram, cpu, i/o...).
>>
>>     -- 
>>
>>     Philippe Blouin,
>>     Directeur de la technologie
>>
>>     Tél.  : (833) 465-4276, poste 230
>>     philippe.blouin at inLibro.com <mailto:philippe.blouin at inLibro.com>
>>
>>     inLibro| pour esprit libre |www.inLibro.com <http://www.inLibro.com>
>>
>     _______________________________________________
>     Koha-devel mailing list
>     Koha-devel at lists.koha-community.org
>     https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
>     website : https://www.koha-community.org/
>     git : https://git.koha-community.org/
>     bugs : https://bugs.koha-community.org/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.koha-community.org/pipermail/koha-devel/attachments/20221221/23c100d6/attachment-0001.htm>