https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on|27344 |30172 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27344 [Bug 27344] Implement Elastic's update_index_background using Koha::BackgroundJob https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30172 [Bug 30172] Background jobs failing due to race condition -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart+koha@gmail. | |com, | |nick@bywatersolutions.com, | |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |martin.renvoize@ptfs-europe |ity.org |.com -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au --- Comment #3 from David Cook <dcook@prosentient.com.au> --- I don't know about this one. The database fall-back already doesn't scale, so this would make it so that you could only ever have a maximum of 1 background_jobs_worker.pl for a queue. For this particular use case, wouldn't it be better to enqueue any tasks that weren't enqueued due to RabbitMQ being offline? (To avoid a race condition, you could either have 1 process responsible for that activity, or you could use database locking I suppose. I'd have to double-check the Koha::BackgroundJob code.) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 --- Comment #5 from Tomás Cohen Arazi <tomascohen@gmail.com> --- We need a PID column. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |In Discussion -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 --- Comment #8 from David Cook <dcook@prosentient.com.au> --- (In reply to Martin Renvoize from comment #0)

Our worker currently starts up and immediately tries to listen for jobs being passed via STOMP. However, if rabbitMQ wasn't running when the tasks were enqueue, then the worker will never know about them.

We should work through the outstanding queue before listening for new jobs.

I was thinking about this again as I'm building a RabbitMQ based job queue for a different Perl project. (Part of me thought maybe I should just use Minion, but the Perl and DB dependencies aren't available and I'm already using RabbitMQ for other asynchronous work on that system.) It seems to me that the order of operations should be the following: 1. Connect to RabbitMQ 2. Insert job in DB 3. Commit job in DB 4. Send message to a durable RabbitMQ queue 5. Output result of send message to user 5a. If successful, say job has been created 5b. If unsuccessful, say there was a problem creating job and to contact an administrator -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 --- Comment #10 from David Cook <dcook@prosentient.com.au> --- (In reply to David Cook from comment #8)

It seems to me that the order of operations should be the following:

1. Connect to RabbitMQ 2. Insert job in DB 3. Commit job in DB 4. Send message to a durable RabbitMQ queue 5. Output result of send message to user 5a. If successful, say job has been created 5b. If unsuccessful, say there was a problem creating job and to contact an administrator

Actually that should be: 1. Connect to RabbitMQ 2. Insert job in DB 3. Commit job in DB 4. Send message to a durable RabbitMQ queue 5. Output result of send message to user 5a. If successful, say job has been created. 5b. If unsuccessful, say there was a problem creating job and to contact an administrator, and update/commit job in DB as failed. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=30889 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 Tomás Cohen Arazi <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kyle@bywatersolutions.com, | |liz@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 --- Comment #14 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi from comment #12)

Let's kill the rabbit ;-)

I won't oppose it. Minion might make more sense after all. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 --- Comment #16 from David Cook <dcook@prosentient.com.au> --- (In reply to Tomás Cohen Arazi from comment #13)

Send message to a durable RabbitMQ queue(In reply to David Cook from comment #10)

...

4. Send message to a durable RabbitMQ queue

What is the 'durable RabbitMQ queue' in this context?

"Durable (the queue will survive a broker restart)" (https://www.rabbitmq.com/queues.html)

In order for using RabbitMQ makes sense, I feel like we need:

- A Task queue manager (we have the table, just missing a process that polls it and sends 'the message' - A way to dispatch the message to notify workers about it (Yay, RabbitMQ) - A worker that reads the message and acts accordingly

Questions:

- how does a worker report back if it failed? Probably directly to the DB? This is a case for a task queue manager, which would read and make the decision it is time to retry. - if we had 2 workers waiting for 'index' jobs through the mq, how do we pick which process takes the job? how does the system know it is being processed by who? what if the process dies and we need to assign a new worker? That's where I was going with the PID thing. BEcause I was thinking about things locally (wrong) but the case still stands with this question. Who is running what?

- If a worker fails, it needs to record that in the "result store" (which in this case is the database). - Why do you need to know which worker processes which message? It shouldn't matter. If the worker dies, the MQ broker would detect the broken connection, and give that message to the next worker that connects asking for a message.

I personally don't see a case in which RabbitMQ is being useful right now. At least when we still haven't implemented a way to avoid tasks that failed to get notified through the mq to be retried, scheduled, etc.

It's not the job of the message broker to handle scheduling. There needs to be a separate scheduler. That can be a cronjob, it can be a daemon. It's whatever we want it to be. (The advantage of the minion daemon is that it already exists and we wouldn't need to write it.) Are you having problems with retries? RabbitMQ should automatically retry. Or are you referring to this scenario: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654#c0 RabbitMQ is useful right now because it provides a standard interface for passing message between separate processes. It's the only reason we're able to process asynchronous jobs in a reasonable way. (Forking CGI processes for the old school background jobs has a major barrier to using Plack 100%.) (In reply to Martin Renvoize from comment #0)

Our worker currently starts up and immediately tries to listen for jobs being passed via STOMP. However, if rabbitMQ wasn't running when the tasks were enqueue, then the worker will never know about them.

Why wouldn't RabbitMQ be running? I'd argue that we shouldn't be saving jobs in the database if the message broker (which is a component of the overall Koha system) is unavailable. To me, it sounds like a Koha design problem rather than a RabbitMQ problem. If the database weren't available, we wouldn't try saving a change to the file system on our own. We'd throw an exception saying that we can't save the record. I'm in the process of implementing a background job system using RabbitMQ on another Perl app and that's the process I'll be following. First I'll check my RabbitMQ connection, if good, then I'll insert the job into the DB, commit into the DB, which is necessary to avoid a race condition, send the message to RabbitMQ. If the message sends, then there's nothing more to do but tell the user that their job is in progress. If it doesn't send, I update the job in the database as failed (or you could delete the job), and tell the user that there was an error. I'm not saying that's necessarily the only right way to do it. Or that we have to keep RabbitMQ, but that's my interpretation of the situation. Totally not opposed to yanking out RabbitMQ and replacing with Minion. I think that we'd gain a lot of functionality for free by doing that, but I suppose I want to make sure we're not unfairly condemning RabbitMQ either when I think we're the reason RabbitMQ might not be working the way we want it to. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 --- Comment #18 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #17)

(In reply to David Cook from comment #14)

...

(In reply to Tomás Cohen Arazi from comment #12)

...

Let's kill the rabbit ;-)

I won't oppose it. Minion might make more sense after all.

Or, as time is limited for everybody, we take advantage of having something that is working (and has been waiting for years), a task queue (whatever is its implementation), and fix the different scripts that are still running under cgi?

We have been waiting for it to adjust the scripts. Now we have it we would like to rethink/rewrite everything? What's the point for the end-users exactly? Bug 27421 is one step forward, and does not get attention.

Could we show durability in our tech choices?

I've already answered several about why rabbitmq.

I am out of the recurring discussion anyway, just wanted to say that.

That makes sense to me too. I'm happy to support whomever has the time to do the work one way or another. (I've already given feedback on Bug 27421 but I'm happy to look at it again if it's ready to be QAed.) -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30654 Martin Renvoize <martin.renvoize@ptfs-europe.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |WONTFIX Status|In Discussion |RESOLVED -- You are receiving this mail because: You are watching all bug changes.