[Bug 32573] New: background_jobs_worker.pl should ACK a message before it forks and runs the job
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Bug ID: 32573 Summary: background_jobs_worker.pl should ACK a message before it forks and runs the job Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: kyle@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org Splitting off this functionality from bug 32558. This is the comment that started this discussion: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32558#c15
From the O'Reilly book Mobile and Web Messaging:
By default, the STOMP broker will consider the message automatically acknowledged when it is delivered to the consumer. However, there are cases in which the consumer may prefer to explicitly acknowledge the message. It leaves a window of opportunity to determine whether it can handle the message or not. For example, the client needs to write the message payload in a data store. There may be issues with opening a connection to the data store and the client could choose to acknowledge the message only after having successfully written its body to the data store. In case of failure, it will instead nack the message (explicitly refuse to take ownership of it). When the STOMP broker is informed of this negative acknowledgment, it may then decide to deliver the message to another consumer subscribed to the destination or try again some time later depending on its configuration. ---
--- RabbitMQ STOMP plugin supports auto, client, and client-individual subscription headers that affect how ACK on NACK operations work. The auto mode uses automatic acknowledgements. The client mode is manual (client-driven) acknowledgements of multiple messages at once. The client-individual is for message-by-message manual acknowledgement. --- If ACK is meant to be sent post-work, why would automatic acknowledgement exist? The interpretation that an ACK is meant to be sent after handling the processing of a job doesn't make sense. In summary, we should be sending back ACK messages as soon as we've successfully processed ( e.g. read and decoded the data ) of a frame. Only then should we process it. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=32558 -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |32558 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32558 [Bug 32558] Allow background_jobs_worker.pl to process multiple jobs simultaneously up to a limit -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |kyle@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #1 from Kyle M Hall <kyle@bywatersolutions.com> --- Created attachment 145066 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145066&action=edit Bug 32573: Send ACK message to RabbitMQ before handling the job Splitting off this functionality from bug 32558. This is the comment that started this discussion: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32558#c15
From the O'Reilly book Mobile and Web Messaging:
By default, the STOMP broker will consider the message automatically acknowledged when it is delivered to the consumer. However, there are cases in which the consumer may prefer to explicitly acknowledge the message. It leaves a window of opportunity to determine whether it can handle the message or not. For example, the client needs to write the message payload in a data store. There may be issues with opening a connection to the data store and the client could choose to acknowledge the message only after having successfully written its body to the data store. In case of failure, it will instead nack the message (explicitly refuse to take ownership of it). When the STOMP broker is informed of this negative acknowledgment, it may then decide to deliver the message to another consumer subscribed to the destination or try again some time later depending on its configuration. ---
--- RabbitMQ STOMP plugin supports auto, client, and client-individual subscription headers that affect how ACK on NACK operations work. The auto mode uses automatic acknowledgements. The client mode is manual (client-driven) acknowledgements of multiple messages at once. The client-individual is for message-by-message manual acknowledgement. --- If ACK is meant to be sent post-work, why would automatic acknowledgement exist? The interpretation that an ACK is meant to be sent after handling the processing of a job doesn't make sense. In summary, we should be sending back ACK messages as soon as we've successfully processed ( e.g. read and decoded the data ) of a frame. Only then should we process it. Test Plan: 1) Apply this patch 2) Launch background_jobs_worker.pl 3) Note no changes in background job processing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |emily.lamancusa@montgomeryc | |ountymd.gov -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart+koha@gmail. | |com, | |kyle@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #2 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- I've tried this when working on bug 32481 but it didn't help. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #3 from David Cook <dcook@prosentient.com.au> --- I don't think there's 100% perfect answer for this topic. If you look at "Message acknowledgement" on the following work queue tutorial, it explicitly says that you should ACK after the work task has completed. It does note how the 30 minute default timeout can be an issue. https://www.rabbitmq.com/tutorials/tutorial-two-python.html "Doing a task can take a few seconds, you may wonder what happens if a consumer starts a long task and it terminates before it completes. With our current code once RabbitMQ delivers message to the consumer, it immediately marks it for deletion. In this case, if you terminate a worker, the message it was just processing is lost. The messages that were dispatched to this particular worker but were not yet handled are also lost. But we don't want to lose any tasks. If a worker dies, we'd like the task to be delivered to another worker. In order to make sure a message is never lost, RabbitMQ supports message acknowledgments. An ack(nowledgement) is sent back by the consumer to tell RabbitMQ that a particular message had been received, processed and that RabbitMQ is free to delete it. If a consumer dies (its channel is closed, connection is closed, or TCP connection is lost) without sending an ack, RabbitMQ will understand that a message wasn't processed fully and will re-queue it. If there are other consumers online at the same time, it will then quickly redeliver it to another consumer. That way you can be sure that no message is lost, even if the workers occasionally die." -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #4 from David Cook <dcook@prosentient.com.au> --- If you were doing a big batch record modification and you suffered a power outage on the machine where the background job worker is running, then that task can't be re-started since the RabbitMQ message was already ACKed before the real work began. (Of course, in the case of Koha, we have the "database failover", so in theory it might get recovered, but it's not standard design.) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #5 from Kyle M Hall <kyle@bywatersolutions.com> --- (In reply to David Cook from comment #4)
If you were doing a big batch record modification and you suffered a power outage on the machine where the background job worker is running, then that task can't be re-started since the RabbitMQ message was already ACKed before the real work began.
(Of course, in the case of Koha, we have the "database failover", so in theory it might get recovered, but it's not standard design.)
Considering batch record modifications are not atomic, I don't think that's necessarily a compelling argument. I can't see any scenario where either: A) We should ack before processing. We are stymying Koha's ability to do background processing by running extrememly long processes in serial with no way to fork multiple background jobs at one. B) RabbitMQ/STOMP is the wrong tool for the job and we should remove it's use from Koha. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #6 from Kyle M Hall <kyle@bywatersolutions.com> ---
I can't see any scenario where either: A) We should ack before processing. We are stymying Koha's ability to do background processing by running extrememly long processes in serial with no way to fork multiple background jobs at one.
B) RabbitMQ/STOMP is the wrong tool for the job and we should remove it's use from Koha.
Actually, I can revise A a bit. We can fork and give the existing rabbit connection to the child process, and have the parent start a new connection to pass off again. That mains we can parallel process without acking first. It doesn't solve the 30 minutes time limit ( which is why I'm going to continue advocating for acking first ) but it does improve the situation greatly. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nick@bywatersolutions.com --- Comment #7 from Nick Clemens <nick@bywatersolutions.com> --- I think for Koha we should ack as soon as we receive the job from RabbitMQ We maintain our own DB table for background jobs, and this is our source of truth. We update this table with the results of a job, and use this table to determine status. If we maintained Rabbit as the source of truth, we might want to keep jobs there until processed, but would then need to check Rabbit for the job statuses Most of the jobs we are sending won't benefit from a retry if they fail - a bad record won't index no matter how many times we try. If we have jobs that are network based, and may fail because of connection issues, we should handle the re-queuing in the job itself - so ack the job - try it, if fail then queue it as new - then we can note the failure/requeue in Koha as well It looks like the Stomp protocol recommendations differ from the Net::Stomp and RabbitMQ docs, but I also don't see a negative side effect of ack before process -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- (In reply to Kyle M Hall from comment #5)
Considering batch record modifications are not atomic, I don't think that's necessarily a compelling argument.
I think that shows that batch modifications are problematically designed. Consider my comment on bug 32278 that talks about how having item/record numbers saved into tracked batches. We could then process those batches in an atomic fashion.
I can't see any scenario where either: A) We should ack before processing. We are stymying Koha's ability to do background processing by running extrememly long processes in serial with no way to fork multiple background jobs at one.
B) RabbitMQ/STOMP is the wrong tool for the job and we should remove it's use from Koha.
Actually, I can revise A a bit. We can fork and give the existing rabbit connection to the child process, and have the parent start a new connection to pass off again. That mains we can parallel process without acking first. It doesn't solve the 30 minutes time limit ( which is why I'm going to continue advocating for acking first ) but it does improve the situation greatly.
From day one, I've advocated for multiple worker processes. Frequently, you start multiple worker processes independently of the web app. I have other applications where I run the worker processes in standalone containers and I can start as many as I want to do the work based on the workload of that application (based on the computational power I have available of course). I also put them together in pipelines which means I can process lots of data very quickly. Those apps are way more data heavy than Koha. I still ACK after I process my RabbitMQ messages.
In Koha, we shouldn't be having extremely long running processes. We don't do any 1 thing that is really computationally that heavy; we're not running complex image manipulation on high resolution medical images with terabytes worth of data. We're just writing inefficient suboptimal code. RabbitMQ isn't the wrong tool for the job, but I think the Koha community doesn't have the knowledge/experience to know how to use it (well). And I'll include myself in that camp. I think that I've probably used RabbitMQ more than anyone else in the Koha community, and I'd still consider myself a RabbitMQ newbie. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- (In reply to Nick Clemens from comment #7)
I think for Koha we should ack as soon as we receive the job from RabbitMQ
We maintain our own DB table for background jobs, and this is our source of truth. We update this table with the results of a job, and use this table to determine status.
As you say, it's the "result store". As far as task queues go, they typically have a database for a result store. That is separate from the messaging layer.
If we maintained Rabbit as the source of truth, we might want to keep jobs there until processed, but would then need to check Rabbit for the job statuses
RabbitMQ is the messaging layer. It's not a data layer.
Most of the jobs we are sending won't benefit from a retry if they fail - a bad record won't index no matter how many times we try. If we have jobs that are network based, and may fail because of connection issues, we should handle the re-queuing in the job itself - so ack the job - try it, if fail then queue it as new - then we can note the failure/requeue in Koha as well
If a bad record won't index, then you fail the job. That still has nothing to do with the messaging layer. Once you fail the job and record that in the result store, you ack the message to tell the messaging layer that you've handled that message. In terms of re-queuing, Koha doesn't really support that at the moment, since there is no task scheduler. You can't re-queue for 5 minutes later to see if the network has stabilized. That would be future functionality.
It looks like the Stomp protocol recommendations differ from the Net::Stomp and RabbitMQ docs, but I also don't see a negative side effect of ack before process
Can you refer to that recommendation specifically? I'm not sure what you mean. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- But... as I note at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27783#c64 my libraries are not heavy users of background tasks at this point. It sounds like Bywater are having the most struggles with the limitations of current background job implementation, and probably with the batch record modification? With the current code, I can see how that could be a problem. It would probably take quite a bit of re-engineering to break up a batch record modification into chunks that could be easily digested. So I have a compromise... Celery has an "acks_late" option for its tasks. Why don't we implement that option on a per-task basis? For tasks that are idempotent, we can keep them with acks_late like we have now. But for tasks that are more likely to run long (like the batch mods), then we could ack early? That would be fairly trivial to code. -- For ack early jobs, we could also have a cronjob that marks them as failed after a configurable time (i.e. a time much longer than 30 minutes). Like Minion's stuck_after: https://metacpan.org/pod/Minion#stuck_after -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tomascohen@gmail.com --- Comment #11 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to David Cook from comment #8)
RabbitMQ isn't the wrong tool for the job
The question is: are we better with just polling? What do we lose? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |m.de.rooy@rijksmuseum.nl -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #12 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi from comment #11)
(In reply to David Cook from comment #8)
RabbitMQ isn't the wrong tool for the job
The question is: are we better with just polling? What do we lose?
For starters, I'd refer people to Jonathan's comment from July: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654#c17 Database polling has a number of shortcomings. It's slower and less efficient. It's not easily scalable. You have to implement all the messaging and concurrent processing logic from scratch. It's a naïve method of queue processing that people resort to because they understand relational databases and don't understand message queues. I think that one could argue that Koha is allowed to be slow, inefficient, and not scalable. The amount of background processing that Koha needs to do is going to be minor in comparison to many other systems. So I imagine that Koha could get along just fine with a bespoke database polling background job processing system. In hindsight, we probably should've done that originally. Perhaps we were too ambitious in our desire to use modern tools and standards. But I think that just using database polling would be a backwards step for the project overall at this point. Koha's usage of RabbitMQ is improving. We devs are collectively improving. I think we really only have 2 remaining problems: 1. How to handle jobs that run longer than the RabbitMQ default 30 minute ack timeout (I think the "ack_late" for some jobs is a good compromise to handle this - alternatively Koha implementors can tune their own RabbitMQ installations if they find the default is too low) 2. How to have multiple workers when using Koha packages (this is easy with RabbitMQ, but the existing database polling failover makes this significantly harder, but it's a solvable problem) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #13 from David Cook <dcook@prosentient.com.au> --- I was looking back at the origin of the database failover, and I think Julian makes an interesting point at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417#c249 Maybe we could offer two options via configuration. Database polling or RabbitMQ. However, for database polling, we should try to provide at least a minimum of similar functionality. That is to say, "misc/background_jobs_worker.pl" would just be a worker. Instead of connecting to RabbitMQ, it could connect to our own bespoke job server. That bespoke job server could use an event loop to poll the database for jobs and check that it has workers that can accept jobs. It's not that hard to write that code. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |In Discussion --- Comment #14 from David Cook <dcook@prosentient.com.au> --- Regarding the subject of early ACK and late ACK, consider the following RabbitMQ discussion between a user and 2 key RabbitMQ devs (both I've spoken to personally in the past): https://github.com/rabbitmq/rabbitmq-server/discussions/4030 Luke's recommendation is to update the global consumer timeout. Michael mentions most consumers shouldn't need longer than 30 minutes to process a message, but for operations that are expected to take longer than 30 minutes - you can ACK early and use a separate progress metric. Since we're using the database result store, end users can use that as a progress metric. And if we have a cronjob that runs periodically, we can evaluate the start time and fail tasks that run longer than our own configurable timeout. I think that would resolve this ACK issue perfectly. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #15 from David Cook <dcook@prosentient.com.au> --- I'm curious what Kyle thinks about it. Whether he wants to update his patches, or if folk want me to provide alternative patches. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #16 from David Cook <dcook@prosentient.com.au> --- Another thought... What about imposing limits on the size of batches? It's common to see this on major commercial systems. Some social media sites will only allow you to choose 50 items for an operation at a time. Related to that would be breaking up operations into multiple batches but I know that's easier said than done at this point. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #17 from Tomás Cohen Arazi <tomascohen@gmail.com> --- (In reply to Tomás Cohen Arazi from comment #11)
(In reply to David Cook from comment #8)
RabbitMQ isn't the wrong tool for the job
The question is: are we better with just polling? What do we lose?
I didn't intend to restart this conversation. Just to start over on which benefits it offers and why we should go full MQ. I think we never entirely implement it correctly. Regarding the ack discussion, it feels to me that many of our problems are related to the 'ack last' behavior. As you said, we have our own way of tracking progress and termination, so we are safe to just ack and leave that to Koha. I've been pondering having separate workers for polling and rabbit respectively. It would simplify things a lot. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #18 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi from comment #17)
Regarding the ack discussion, it feels to me that many of our problems are related to the 'ack last' behavior. As you said, we have our own way of tracking progress and termination, so we are safe to just ack and leave that to Koha.
Well, I think the end user can track the progress, but we don't currently have a systematic way of catching "stuck" jobs. But that's a failure scenario that should hopefully be rarer than consumer acknowledgement timeouts. We'll get there. I'm curious to hear what Michael and Luke from RabbitMQ advise...
I've been pondering having separate workers for polling and rabbit respectively. It would simplify things a lot.
That does sound interesting. Could you explain that one more? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #19 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #18)
I've been pondering having separate workers for polling and rabbit respectively. It would simplify things a lot.
That does sound interesting. Could you explain that one more?
Actually, I think that I might know what you mean. If we had separate processes for polling and rabbit, people who prefer polling could use polling, and people who prefer rabbit could use rabbit. That could make things easier, as concurrent processing for polling and rabbit needs to be handled differently. And acknowledgements may or may not be used with polling... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #20 from Kyle M Hall <kyle@bywatersolutions.com> --- Nick asked me to post this https://stomp.github.io/stomp-specification-1.2.html#ACK --- ACK is used to acknowledge consumption of a message from a subscription using client or client-individual acknowledgment. Any messages received from such a subscription will not be considered to have been consumed until the message has been acknowledged via an ACK. --- The stomp protocol doesn't specify that an ACK is meant to indicate the completion of whatever activity a message would require. An ACK is only meant to verify that the client received the message. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #21 from Kyle M Hall <kyle@bywatersolutions.com> --- (In reply to David Cook from comment #19)
(In reply to David Cook from comment #18)
I've been pondering having separate workers for polling and rabbit respectively. It would simplify things a lot.
That does sound interesting. Could you explain that one more?
Actually, I think that I might know what you mean. If we had separate processes for polling and rabbit, people who prefer polling could use polling, and people who prefer rabbit could use rabbit.
That could make things easier, as concurrent processing for polling and rabbit needs to be handled differently. And acknowledgements may or may not be used with polling...
I get where you are coming from, but the failover to polling has definitely provided valuable. If we ACK first, it's not too difficult to handle both in one script. If we continue to ACK last, it's more complicated but still possible. I'd say we need to make the decision on where to ACK and the rest of the plan will follow from that. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #22 from David Cook <dcook@prosentient.com.au> --- (In reply to Kyle M Hall from comment #20)
Nick asked me to post this
https://stomp.github.io/stomp-specification-1.2.html#ACK --- ACK is used to acknowledge consumption of a message from a subscription using client or client-individual acknowledgment. Any messages received from such a subscription will not be considered to have been consumed until the message has been acknowledged via an ACK. ---
The stomp protocol doesn't specify that an ACK is meant to indicate the completion of whatever activity a message would require. An ACK is only meant to verify that the client received the message.
The spec doesn't seem to strictly define "consume". I imagine that they've left it intentionally ambiguous so that they don't prescribe the behaviour of message consumers. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #23 from David Cook <dcook@prosentient.com.au> --- (In reply to Kyle M Hall from comment #21)
I'd say we need to make the decision on where to ACK and the rest of the plan will follow from that.
Jonathan has put in the early ACK into bug 32393 at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32393#c24 I don't know that adding it to that particular patch set is the best idea, but I'll leave it up to you guys to sort it out. I'm not a heavy user of RabbitMQ in Koha at this point, so happy for you folk to make the decision and do the work. If it's an issue later, we can deal with it later. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=32393 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #24 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- See also 32393. It also moves the ack before processing the job. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |32393 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32393 [Bug 32393] background job worker explodes if JSON is incorrect -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Discussion |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145066|0 |1 is obsolete| | --- Comment #25 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 145594 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145594&action=edit Bug 32573: Send ACK message to RabbitMQ before handling the job Splitting off this functionality from bug 32558. This is the comment that started this discussion: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32558#c15
From the O'Reilly book Mobile and Web Messaging:
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145594|0 |1 is obsolete| | --- Comment #26 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 145595 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145595&action=edit Bug 32573: Send ACK message to RabbitMQ before handling the job Splitting off this functionality from bug 32558. This is the comment that started this discussion: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32558#c15
From the O'Reilly book Mobile and Web Messaging:
By default, the STOMP broker will consider the message automatically acknowledged when it is delivered to the consumer. However, there are cases in which the consumer may prefer to explicitly acknowledge the message. It leaves a window of opportunity to determine whether it can handle the message or not. For example, the client needs to write the message payload in a data store. There may be issues with opening a connection to the data store and the client could choose to acknowledge the message only after having successfully written its body to the data store. In case of failure, it will instead nack the message (explicitly refuse to take ownership of it). When the STOMP broker is informed of this negative acknowledgment, it may then decide to deliver the message to another consumer subscribed to the destination or try again some time later depending on its configuration. ---
--- RabbitMQ STOMP plugin supports auto, client, and client-individual subscription headers that affect how ACK on NACK operations work. The auto mode uses automatic acknowledgements. The client mode is manual (client-driven) acknowledgements of multiple messages at once. The client-individual is for message-by-message manual acknowledgement. --- If ACK is meant to be sent post-work, why would automatic acknowledgement exist? The interpretation that an ACK is meant to be sent after handling the processing of a job doesn't make sense. In summary, we should be sending back ACK messages as soon as we've successfully processed ( e.g. read and decoded the data ) of a frame. Only then should we process it. Test Plan: 1) Apply this patch 2) Launch background_jobs_worker.pl 3) Note no changes in background job processing Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |m.de.rooy@rijksmuseum.nl |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |BLOCKED --- Comment #27 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- QAing -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|BLOCKED |Passed QA Patch complexity|--- |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145595|0 |1 is obsolete| | --- Comment #28 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 145608 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145608&action=edit Bug 32573: Send ACK message to RabbitMQ before handling the job Splitting off this functionality from bug 32558. In summary, we should be sending back ACK messages as soon as we've successfully processed ( e.g. read and decoded the data ) of a frame. Only then should we process it. Test plan: 1) Apply this patch 2) Launch background_jobs_worker.pl 3) Note no changes in background job processing Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #29 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Had an issue with the commit message being truncated. I only kept the summary now, the arguments from O'Reilly etc. are on this report. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #145608|0 |1 is obsolete| | --- Comment #30 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- Created attachment 145612 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=145612&action=edit Bug 32573: Send ACK message to RabbitMQ before handling the job Splitting off this functionality from bug 32558. This is the comment that started this discussion: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32558#c15
From the O'Reilly book Mobile and Web Messaging:
By default, the STOMP broker will consider the message automatically acknowledged when it is delivered to the consumer. However, there are cases in which the consumer may prefer to explicitly acknowledge the message. It leaves a window of opportunity to determine whether it can handle the message or not. For example, the client needs to write the message payload in a data store. There may be issues with opening a connection to the data store and the client could choose to acknowledge the message only after having successfully written its body to the data store. In case of failure, it will instead nack the message (explicitly refuse to take ownership of it). When the STOMP broker is informed of this negative acknowledgment, it may then decide to deliver the message to another consumer subscribed to the destination or try again some time later depending on its configuration. ---
--- RabbitMQ STOMP plugin supports auto, client, and client-individual subscription headers that affect how ACK on NACK operations work. The auto mode uses automatic acknowledgements. The client mode is manual (client-driven) acknowledgements of multiple messages at once. The client-individual is for message-by-message manual acknowledgement. --- If ACK is meant to be sent post-work, why would automatic acknowledgement exist? The interpretation that an ACK is meant to be sent after handling the processing of a job doesn't make sense. In summary, we should be sending back ACK messages as soon as we've successfully processed ( e.g. read and decoded the data ) of a frame. Only then should we process it. Test Plan: 1) Apply this patch 2) Launch background_jobs_worker.pl 3) Note no changes in background job processing Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #31 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Marcel de Rooy from comment #29)
Had an issue with the commit message being truncated. I only kept the summary now, the arguments from O'Reilly etc. are on this report.
Reattached with the commit message. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)| |23.05.00 released in| | Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #32 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 23.05. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|23.05.00 |23.05.00,22.11.03 released in| | Status|Pushed to master |Pushed to stable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 --- Comment #33 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- Nice work everyone! Pushed to stable for 22.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Lucas Gass <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|23.05.00,22.11.03 |23.05.00,22.11.03,22.05.10 released in| | Status|Pushed to stable |Pushed to oldstable CC| |lucas@bywatersolutions.com --- Comment #34 from Lucas Gass <lucas@bywatersolutions.com> --- Backported to 22.05.x for upcoming 22.05.10 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 Arthur Suzuki <arthur.suzuki@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|23.05.00,22.11.03,22.05.10 |23.05.00,22.11.03,22.05.10, released in| |21.11.16 CC| |arthur.suzuki@biblibre.com Status|Pushed to oldstable |Pushed to oldoldstable --- Comment #35 from Arthur Suzuki <arthur.suzuki@biblibre.com> --- applied to 21.11.x for 21.11.16 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32573 wainuiwitikapark@catalyst.net.nz changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wainuiwitikapark@catalyst.n | |et.nz --- Comment #36 from wainuiwitikapark@catalyst.net.nz --- Not backported to 21.05.x -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org