[Bug 34990] New: Backgroundjob->enqueue does not send persistent header
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Bug ID: 34990 Summary: Backgroundjob->enqueue does not send persistent header Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Architecture, internals, and plumbing Assignee: koha-bugs@lists.koha-community.org Reporter: m.de.rooy@rijksmuseum.nl QA Contact: testopia@bugs.koha-community.org Restarting RabbitMQ will make us loose all pending messages (jobs) in the default or long_tasks queue. Although we have a database fallback, this might be a problem when e.g. replacing/restarting a container while the workers did not yet read all messages or were down already, and the restart would activate Rabbit first. Note that this scenario places RabbitMQ in the same container; it surely would be better to put it in its own container. But also in that scenario we can loose messages if we do not add a persistent flag. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED Assignee|koha-bugs@lists.koha-commun |m.de.rooy@rijksmuseum.nl |ity.org | -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au, | |jonathan.druart+koha@gmail. | |com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Trivial patch Status|ASSIGNED |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 --- Comment #1 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Created attachment 156564 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=156564&action=edit Bug 34990: Add persistent header when sending msg to RabbitMQ Test plan: NOTE: It is very hard to add a Koha unit test for adding this single header when communicating to RabbitMQ via Stomp plugin. When we would mock the send, we are only testing if perl can pass a hashref to a subroutine ;) Do NOT yet apply this patch. Make sure that RabbitMQ runs. Stop the koha-worker for long_tasks: koha-worker --stop --queue long_tasks myclone Stage a MARC file. Check queues with rabbitmqctl list_queues. Look for: koha_myclone-long_tasks 1 (at least 1) Stop rabbitmq (something like /etc/init.d/rabbitmq-server stop) Start rabbitmq (/etc/init.d/rabbitmq-server start) Check queue again with with rabbitmqctl list_queues. Look for: koha_myclone-long_tasks 0 Your messages have been gone. Now apply this patch. Reiterate the former steps. But note that you will still see a non-empty queue in the last step: koha_myclone-long_tasks 1 (at least 1) Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #156564|0 |1 is obsolete| | --- Comment #2 from David Nind <david@davidnind.com> --- Created attachment 156579 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=156579&action=edit Bug 34990: Add persistent header when sending msg to RabbitMQ Test plan: NOTE: It is very hard to add a Koha unit test for adding this single header when communicating to RabbitMQ via Stomp plugin. When we would mock the send, we are only testing if perl can pass a hashref to a subroutine ;) Do NOT yet apply this patch. Make sure that RabbitMQ runs. Stop the koha-worker for long_tasks: koha-worker --stop --queue long_tasks myclone Stage a MARC file. Check queues with rabbitmqctl list_queues. Look for: koha_myclone-long_tasks 1 (at least 1) Stop rabbitmq (something like /etc/init.d/rabbitmq-server stop) Start rabbitmq (/etc/init.d/rabbitmq-server start) Check queue again with with rabbitmqctl list_queues. Look for: koha_myclone-long_tasks 0 Your messages have been gone. Now apply this patch. Reiterate the former steps. But note that you will still see a non-empty queue in the last step: koha_myclone-long_tasks 1 (at least 1) Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: David Nind <david@davidnind.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 David Nind <david@davidnind.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |david@davidnind.com --- Comment #3 from David Nind <david@davidnind.com> --- I've signed off, as the test plan works. However, what should happen to the jobs queued (shown under Administration > Jobs)? For me, they are staying there with a status of 'New' and progress 'null/0'. The action options I have are view and cancel. Testing notes (using KTD): 1. Add sudo in front of the commands in the test plan. For myclone, use kohadev. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 --- Comment #4 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to David Nind from comment #3)
I've signed off, as the test plan works.
Thanks
However, what should happen to the jobs queued (shown under Administration > Jobs)? For me, they are staying there with a status of 'New' and progress 'null/0'. The action options I have are view and cancel.
Thats another problem, yes. There is only a workaround for that: stop rabbitmq, restart worker. The worker will now go in 'database mode' and get the jobs that are still new in the db. Since they might be quite old, this is very questionable behavior btw. You mention null/0. I saw that too. That is another small bug. Will open two new reports. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 --- Comment #5 from David Nind <david@davidnind.com> --- Thanks Marcel! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=34997, | |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=34998 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #156579|0 |1 is obsolete| | --- Comment #6 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 156871 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=156871&action=edit Bug 34990: Add persistent header when sending msg to RabbitMQ Test plan: NOTE: It is very hard to add a Koha unit test for adding this single header when communicating to RabbitMQ via Stomp plugin. When we would mock the send, we are only testing if perl can pass a hashref to a subroutine ;) Do NOT yet apply this patch. Make sure that RabbitMQ runs. Stop the koha-worker for long_tasks: koha-worker --stop --queue long_tasks myclone Stage a MARC file. Check queues with rabbitmqctl list_queues. Look for: koha_myclone-long_tasks 1 (at least 1) Stop rabbitmq (something like /etc/init.d/rabbitmq-server stop) Start rabbitmq (/etc/init.d/rabbitmq-server start) Check queue again with with rabbitmqctl list_queues. Look for: koha_myclone-long_tasks 0 Your messages have been gone. Now apply this patch. Reiterate the former steps. But note that you will still see a non-empty queue in the last step: koha_myclone-long_tasks 1 (at least 1) Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Failed QA --- Comment #7 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- I have not tested this patch but I am not sure about the consequences. If the status in the DB does not reflect the actual status of the job I don't think we should push as it. Can you clarify please? -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 --- Comment #8 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Jonathan Druart from comment #7)
I have not tested this patch but I am not sure about the consequences. If the status in the DB does not reflect the actual status of the job I don't think we should push as it.
Can you clarify please?
It sounds like extending the scope of this bug? The fact that we have redundancy and that it may be conflicting, is not caused here and is not resolved here. We resolve the loss of messages here for a restart of RabbitMQ. I agree that the BackgroundJob code needs more attention. Somehow it reflects our fear to switch to a message queue? Limping on two thoughts? There are still several reports open about getting the message queue more stable. I am surprised that we cant resolve that since RabbitMQ is commonly used and for large volumes. So I recommend to push this patch and I will certainly support further improvements in this area. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Jonathan Druart <jonathan.druart+koha@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Failed QA |Passed QA --- Comment #9 from Jonathan Druart <jonathan.druart+koha@gmail.com> --- (In reply to Marcel de Rooy from comment #8)
(In reply to Jonathan Druart from comment #7)
I have not tested this patch but I am not sure about the consequences. If the status in the DB does not reflect the actual status of the job I don't think we should push as it.
Can you clarify please?
It sounds like extending the scope of this bug? The fact that we have redundancy and that it may be conflicting, is not caused here and is not resolved here. We resolve the loss of messages here for a restart of RabbitMQ.
I agree that the BackgroundJob code needs more attention. Somehow it reflects our fear to switch to a message queue? Limping on two thoughts? There are still several reports open about getting the message queue more stable. I am surprised that we cant resolve that since RabbitMQ is commonly used and for large volumes.
So I recommend to push this patch and I will certainly support further improvements in this area.
It's not extending the scope of this bug, it's not introducing another bug. IMO Koha UI should show what has been processed. It's better to reenqueue the job manually than tell users the job failed whereas it succeeded. Just my opinion, feel free to ignoring it. Back to PQA as I don't have more time to dedicate to this, but it should be better in discussion... -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 --- Comment #10 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- (In reply to Jonathan Druart from comment #9)
(In reply to Marcel de Rooy from comment #8)
(In reply to Jonathan Druart from comment #7)
I have not tested this patch but I am not sure about the consequences. If the status in the DB does not reflect the actual status of the job I don't think we should push as it.
Can you clarify please?
It sounds like extending the scope of this bug? The fact that we have redundancy and that it may be conflicting, is not caused here and is not resolved here. We resolve the loss of messages here for a restart of RabbitMQ.
I agree that the BackgroundJob code needs more attention. Somehow it reflects our fear to switch to a message queue? Limping on two thoughts? There are still several reports open about getting the message queue more stable. I am surprised that we cant resolve that since RabbitMQ is commonly used and for large volumes.
So I recommend to push this patch and I will certainly support further improvements in this area.
It's not extending the scope of this bug, it's not introducing another bug. IMO Koha UI should show what has been processed. It's better to reenqueue the job manually than tell users the job failed whereas it succeeded.
Just my opinion, feel free to ignoring it.
Back to PQA as I don't have more time to dedicate to this, but it should be better in discussion...
I never ignore it :) Will still check if I am missing something in our discussion. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |BLOCKED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=32305 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |35092 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35092 [Bug 35092] [OMNIBUS] Remaining background job/worker issues -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 --- Comment #11 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- This should go along with 32305. But still looking at it tomorrow. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |32305 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32305 [Bug 32305] Background worker doesn't check job status when received from rabbitmq -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |nick@bywatersolutions.com |y.org | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Marcel de Rooy <m.de.rooy@rijksmuseum.nl> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|BLOCKED |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 --- Comment #12 from Marcel de Rooy <m.de.rooy@rijksmuseum.nl> --- Added dependency on 32305. See omnibus bug. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to master Version(s)| |23.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 --- Comment #13 from Tomás Cohen Arazi <tomascohen@gmail.com> --- Pushed to master for 23.11. Nice work everyone, thanks! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Fridolin Somers <fridolin.somers@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|23.11.00 |23.11.00,23.05.05 released in| | Status|Pushed to master |Pushed to stable CC| |fridolin.somers@biblibre.co | |m --- Comment #14 from Fridolin Somers <fridolin.somers@biblibre.com> --- Pushed to 23.05.x for 23.05.05 -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|23.11.00,23.05.05 |23.11.00,23.05.05,22.11.12 released in| | Status|Pushed to stable |Pushed to oldstable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 --- Comment #15 from Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> --- Nice work everyone! Pushed to oldstable for 22.11.x -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34990 Bug 34990 depends on bug 32305, which changed state. Bug 32305 Summary: Background worker doesn't check job status when received from rabbitmq https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32305 What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to oldstable |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org