[Bug 30172] New: BackGround jobs failing due to race condition
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Bug ID: 30172 Summary: BackGround jobs failing due to race condition Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org We are receiving reports from production of background jobs that show as 'new' in the database that are not being processed. There are no errors or exceptions relating to enqueueing from the broker We do see in the worker-output.log: Can't call method "process" on an undefined value at /usr/share/koha/bin/background_jobs_worker.pl line 80. In Koha/BackGroundJob.pm: 94 sub enqueue { ... 104 $self->_result->result_source->schema->txn_do( 105 sub { 106 $self->set( 107 { 108 status => 'new', ... 114 } 115 )->store; ... 121 try { 122 my $conn = $self->connect; ... 128 $conn->send_with_receipt( { destination => sprintf("/queue/%s-%s", $namespace, $job_type), body => $json_args } ) 129 or Koha::Exceptions::Exception->throw('Job has not been enqueued'); 130 } catch { ... 138 ); ^^End of transaction What appears to be happening is that: 1 - The transaction is begun 2 - Job info is stored internally, but not committed 3 - the job is being sent to Rabbit 4 - Rabbit attempts to process the job, and fails as it cannot find the data in the DB 3 - The transaction completes We are left with jobs sitting at 'new' with no errors and they are never processed -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |22417 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 [Bug 22417] Add a task queue -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Liz Rea <wizzyrea@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wizzyrea@gmail.com --- Comment #1 from Liz Rea <wizzyrea@gmail.com> --- Noting that the rabbitmq server is external to the Koha frontend, and not on the box like with a standard package install when we see this. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Sarah Cornell <sbcornell@cityofportsmouth.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sbcornell@cityofportsmouth. | |com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart+koha@gmail. | |com --- Comment #2 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Which exact version of Koha? Some fixes from 21.11 haven't been backported, it may come from that. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #3 from David Cook <dcook@prosentient.com.au> --- (In reply to Nick Clemens from comment #0)
What appears to be happening is that: 1 - The transaction is begun 2 - Job info is stored internally, but not committed 3 - the job is being sent to Rabbit 4 - Rabbit attempts to process the job, and fails as it cannot find the data in the DB 3 - The transaction completes
I've had this same problem in non-Koha projects. My background worker was too fast, and would get the message before the monolithic app could commit the data to the database. (Note that in my case I didn't actually *need* to do a database call in my background worker, so I just sidestepped the issue.) (In reply to Jonathan Druart from comment #2)
Which exact version of Koha? Some fixes from 21.11 haven't been backported, it may come from that.
I'm looking at master now, and it looks like it has the same race condition. The "send_with_receipt" should be moved outside and after the "txn_do". -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #4 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 131060 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=131060&action=edit Bug 30172: Prevent race condition when enqueuing a new task As we are sending the job to the rabbitmq before in the transaction, the worker can receive the job to process before the transaction committed. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |jonathan.druart+koha@gmail. |ity.org |com -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #131060|0 |1 is obsolete| | --- Comment #5 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 131061 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=131061&action=edit Bug 30172: Prevent race condition when enqueuing a new task As we are sending the job to the rabbitmq before in the transaction, the worker can receive the job to process before the transaction committed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #6 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Nick suggested me to set the status to "failed" in case the job was not enqueued. I am not sure it's the best we can do. If a job is started but the connection failed, the exception will be thrown but the job will appear in the DB table. If the user already got the error on the UI I don't know if the job should be inserted. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #7 from Nick Clemens <nick@bywatersolutions.com> --- There is another approach we could try. In misc/background_jobs_worker.pl we have: 61 my $job = Koha::BackgroundJobs->find($args->{job_id}); 62 63 process_job( $job, $args ); 64 $conn->ack( { frame => $frame } ); # FIXME depending on success? Perhaps: 61 my $job = Koha::BackgroundJobs->find($args->{job_id}); 62 $conn->nack({frame=>$frame}) unless $job; 63 process_job( $job, $args ); 64 $conn->ack( { frame => $frame } ); # FIXME depending on success? Though could we end up in a loop? The queue should expire eventually - and we could store soemthing in the job about retries when we send nack? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #6)
Nick suggested me to set the status to "failed" in case the job was not enqueued. I am not sure it's the best we can do.
If a job is started but the connection failed, the exception will be thrown but the job will appear in the DB table. If the user already got the error on the UI I don't know if the job should be inserted.
You could test the MQ connection before inserting into the database. But if the MQ connection succeeds, you have to insert into the database before sending the message to the MQ. Otherwise that race condition will persist. If the "send_with_receipt" fails, then I think marking the job as "failed" would be fair. (Alternatively, I suppose you could delete the job after warning the user it failed.) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- (In reply to Nick Clemens from comment #7)
Perhaps: 61 my $job = Koha::BackgroundJobs->find($args->{job_id}); 62 $conn->nack({frame=>$frame}) unless $job; 63 process_job( $job, $args ); 64 $conn->ack( { frame => $frame } ); # FIXME depending on success?
Though could we end up in a loop? The queue should expire eventually - and we could store soemthing in the job about retries when we send nack?
There's a few problems with that suggestion. First, you'd need an if/else rather than an unless. Otherwise, you're nacking, then trying to process the job, and then acking. I imagine that was just a typo though. Second, as you say, you'll end up in a loop if you nack. I don't think the message expires either. (Of course, if the issue is just the race condition, eventually the data should exist, so you'd only loop X times until the database catches up. Not efficient, especially as you'd be blocking other messages being processed, but technically functional.) -- I think the best bet is just to make sure the data is committed to the database before sending the message. (Alternatively, you can make sure that all the data you need is sent in the message itself, so that there is no race condition, but that can be more difficult to manage. In terms of managing the result store, you can generate a UUID, store it in the job table, and use it for storing the result in a result_store table, and then use the UUID to unite the two later, but that would be a big refactor I imagine.) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #10 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Can we get a signoff on this patch? It should fix the problem initially reported here. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #11 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 132023 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132023&action=edit Bug 30172: Ensure we can reach the server before inserting the job -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #12 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- This follow-up will ensure the connection can be established before we insert the job. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|BackGround jobs failing due |Background jobs failing due |to race condition |to race condition -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #13 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Something smells here :-D We have a daemon that runs in two ways: listening to the STOMP protocol, or just polling the DB... And we have a task initiator (i.e. a call to ->enqueue) that needs to decide if it does through the STOMP protocol or just inserts on the DB... It feels like the message broker is only adding complexity with no obvious gain? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #14 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi from comment #13)
We have a daemon that runs in two ways: listening to the STOMP protocol, or just polling the DB...
And we have a task initiator (i.e. a call to ->enqueue) that needs to decide if it does through the STOMP protocol or just inserts on the DB...
Could you re-phrase this? I'm not sure what you're saying here. Ultimately, the database acts as a result store. The message broker passes the message from the web app to a background worker. The background worker processes the message and updates the result store. The web app exposes the result store to web users. The same model is used by other task queues like Celery (https://github.com/celery/celery) and Minion (https://metacpan.org/pod/Minion -> https://metacpan.org/release/SRI/Minion-10.23/source/lib/Minion/Backend/Pg.p...). While Celery uses message brokers, Minion does poll the database. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi from comment #13)
It feels like the message broker is only adding complexity with no obvious gain?
I think the obvious gain is asynchronous background processing (using a language agnostic industry standard message broker). Koha certainly has need for a lot more asynchronous background processing. That said, there are times where I wish that we'd just used Minion instead as it seems to be a complete solution - whereas we've created a partial bespoke solution around RabbitMQ. (That said, Minion also only supports PostgreSQL out of the box. There's a third-party MySQL backend but it looks a little dodgy.) One of my hopes with RabbitMQ was that it would allow us to move beyond just Perl (whereas Minion is Perl-only). However, in practice, I don't think we've needed to branch out beyond Perl server-side. Even if we did, there would still be the issue of the result store. We'd need to either use a HTTP API for recording results or build an interface to the result store in whichever language we were using, which would mean some code duplication. Pros and cons... In any case though, I think this particular bug report is just a growing pain. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #16 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- What about the last patch setw David? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #17 from Tomás Cohen Arazi <tomascohen@gmail.com> --- My comment was not meant to hijack this bug, which is real and needs to get fixed. Testing. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #131061|0 |1 is obsolete| | Attachment #132023|0 |1 is obsolete| | --- Comment #18 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 132140 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132140&action=edit Bug 30172: Prevent race condition when enqueuing a new task As we are sending the job to the rabbitmq before in the transaction, the worker can receive the job to process before the transaction committed. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #19 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 132141 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132141&action=edit Bug 30172: Ensure we can reach the server before inserting the job -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #20 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Patches rebased. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Failed QA --- Comment #21 from Nick Clemens <nick@bywatersolutions.com> --- After these patches no jobs are enqueued if you cannot reach the broker I aded to koha-conf: <message_broker> <hostname>localhost</hostname> <port>61614</port> <username>guest</username> <password>guest</password> <vhost></vhost> </message_broker> Before patches, job added, picked up by SQL polling After patches: Cannot enqueue this job. (The error was: Failed to connect: Error connecting to localhost:61614: Cannot assign requested address at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. . See the Koha logfile for more information). -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #22 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Nick Clemens from comment #21)
After these patches no jobs are enqueued if you cannot reach the broker
I aded to koha-conf: <message_broker> <hostname>localhost</hostname> <port>61614</port> <username>guest</username> <password>guest</password> <vhost></vhost> </message_broker>
Before patches, job added, picked up by SQL polling
After patches: Cannot enqueue this job. (The error was: Failed to connect: Error connecting to localhost:61614: Cannot assign requested address at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. . See the Koha logfile for more information).
That's what I understood... "You could test the MQ connection before inserting into the database. " -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #23 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- If the connection fails, the user gets an error. They know that the job hasn't been enqueued. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #24 from Nick Clemens <nick@bywatersolutions.com> --- (In reply to Jonathan Druart from comment #23)
If the connection fails, the user gets an error. They know that the job hasn't been enqueued.
But we have a backup, we should still enqueue the job, even it isn't processed Things are better without second patch, however, we still mark job 'Failed' if we can't connect - it should remain 'new' - marking as failed should happen in the worker or process_job. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #25 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 132176 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132176&action=edit Bug 30172: Enqueue the job even if rabbitmq is not reachable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #26 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- I forgot the backup! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132141|0 |1 is obsolete| | Attachment #132176|0 |1 is obsolete| | --- Comment #27 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 132177 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132177&action=edit Bug 30172: Enqueue the job even if rabbitmq is not reachable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132140|0 |1 is obsolete| | Attachment #132177|0 |1 is obsolete| | --- Comment #28 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 132181 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132181&action=edit Bug 30172: Prevent race condition when enqueuing a new task As we are sending the job to the rabbitmq before in the transaction, the worker can receive the job to process before the transaction committed. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #29 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 132182 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132182&action=edit Bug 30172: Enqueue the job even if rabbitmq is not reachable Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132181|0 |1 is obsolete| | --- Comment #30 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132196 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132196&action=edit Bug 30172: Prevent race condition when enqueuing a new task As we are sending the job to the rabbitmq before in the transaction, the worker can receive the job to process before the transaction committed. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132182|0 |1 is obsolete| | --- Comment #31 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Created attachment 132197 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132197&action=edit Bug 30172: Enqueue the job even if rabbitmq is not reachable Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |martin.renvoize@ptfs-europe | |.com Status|Signed Off |Passed QA --- Comment #32 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- Agreed that this resolves the race condition and I can't find any regressions now we've resolved the one's pointed out. I too will admit I'm not sure what benefits RabbitMQ brings us.. but that can be for another bug. Passing QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #33 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- (In reply to Martin Renvoize from comment #32)
I too will admit I'm not sure what benefits RabbitMQ brings us.. but that can be for another bug.
I suppose I do understand.. it reduces DB hits by removing the need for DB Polling.. i.e instead of waiting 10s and then polling the DB for jobs.. we sit and wait for a STOMP message to be received.. and then look to the DB if we need to for further detail.. so it does kinda make sense to have both. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #34 from David Cook <dcook@prosentient.com.au> --- (In reply to Martin Renvoize from comment #33)
(In reply to Martin Renvoize from comment #32)
I too will admit I'm not sure what benefits RabbitMQ brings us.. but that can be for another bug.
I suppose I do understand.. it reduces DB hits by removing the need for DB Polling.. i.e instead of waiting 10s and then polling the DB for jobs.. we sit and wait for a STOMP message to be received.. and then look to the DB if we need to for further detail.. so it does kinda make sense to have both.
+1 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #35 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- We are using a message broker to: * not rewrite our own task scheduler * not overload the server (only 1 task at a time, with the default configuration) * handle load balancing * run some tasks in the time slot you wish (not implement yet of course but that was a plan) * more flexibility: have several workers, running on different servers, each could deal with a given job type Also, we are not strongly tight to RabbitMQ and you are free to provide an alternative implementation (it's less than 20 lines of code). We chose RabbitMQ as it's a reference, light, and working out of the box. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132196|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #132197|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #36 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132241 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132241&action=edit Bug 30172: Prevent race condition when enqueuing a new task As we are sending the job to the rabbitmq before in the transaction, the worker can receive the job to process before the transaction committed. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #37 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Created attachment 132242 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=132242&action=edit Bug 30172: Enqueue the job even if rabbitmq is not reachable Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30345 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30345 [Bug 30345] Koha::BackgroundJob->enqueue should set borrowernumber=undef if no userenv -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30350 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30350 [Bug 30350] Do not hardcode job types -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |martin.renvoize@ptfs-europe |y.org |.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Bug 30172 depends on bug 22417, which changed state. Bug 22417 Summary: Add a task queue https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Owen Leonard <oleonard@myacpl.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Patch doesn't apply -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Patch doesn't apply |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle@bywatersolutions.com --- Comment #38 from Kyle M Hall <kyle@bywatersolutions.com> --- Doesn't apply to 21.11.x. Please rebase if you'd like this to be backported. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to master |Pushed to stable Version(s)| |,21.11.05 released in| | --- Comment #39 from Kyle M Hall <kyle@bywatersolutions.com> --- Pushed to 21.11.x for 21.11.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|,21.11.05 |22.05.00,21.11.05 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to stable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew@bywatersolutions.com Status|Pushed to stable |Pushed to oldstable Version(s)|22.05.00,21.11.05 |22.05.00,21.11.05,21.05.14 released in| | --- Comment #40 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Pushed to 21.05.x for 21.05.14 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Victor Grousset/tuxayo <victor@tuxayo.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |victor@tuxayo.net Version(s)|22.05.00,21.11.05,21.05.14 |22.05.00,21.11.05,21.05.14, released in| |20.11.18 Status|Pushed to oldstable |Pushed to oldoldstable --- Comment #41 from Victor Grousset/tuxayo <victor@tuxayo.net> --- Backported: Pushed to 20.11.x branch for 20.11.18 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |magnus@libriotech.no --- Comment #42 from Magnus Enger <magnus@libriotech.no> --- Is there a quick way to "kickstart" those jobs that are stuck at "New" in the list of background jobs? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to oldoldstable |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #43 from Martin Renvoize <martin.renvoize@ptfs-europe.com> --- With Rabbit there is no way to do it right now as far as I'm aware... You might be able to kill the worker, then kill rabbit, then start the worker... from memory it will fail to connect to rabbit and fall back to db polling.. that first db polling should find all the new tasks and work through them. Once you've confirmed all your tasks have run you can start rabbit and then restart your worker again to get it to use rabbit instead. We really aught to improve that. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |30654 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 [Bug 30654] Even with RabbitMQ enabled, we should poll the database for jobs at worker startup -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #44 from Magnus Enger <magnus@libriotech.no> --- Right you are, this did the trick: $ sudo koha-worker --stop <instance> $ sudo service rabbitmq-server stop $ sudo koha-worker --start <instance> Thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 --- Comment #45 from Liz Rea <wizzyrea@gmail.com> --- Just wondering what the solution is for external rabbits that you can't stop the rabbitmq server for - disconnect it from kconf? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |januszop@gmail.com --- Comment #46 from Nick Clemens <nick@bywatersolutions.com> --- *** Bug 33287 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org