[Koha-bugs] [Bug 25245] Add a plugin hook to allow running code on a nightly basis

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25245

--- Comment #55 from David Cook <dcook at prosentient.com.au> ---
(In reply to Martin Renvoize from comment #54)
> Yes, a task scheduler would be
> great, but I really didn't want to wait another 10 years whilst people made
> up their mind on how to implement one.

Very fair point.

> 
> Task Schedular != Task Queue
> 

Also very true.

> As for the task queue, now I've looked at it.. I'm rather disappointed that
> it's hardcoded to a list of tasks, doesn't allow for parallelism, doesn't
> offer retries or failure states, doesn't even allow for multiple workers? 

I'm disappointed by the same things, but no one (myself included) was willing
to work on those things. That said, it shouldn't be too hard to refactor (brand
new code...) to have a more dynamic list of potential tasks and using multiple
workers would be fairly trivial to add too. I haven't looked at the code
recently enough to remember how it handles retries/failures, but that's
something that would be fairly app specific anyway I think, so something that
can certainly grow over time.

These are changes I want to make at some point, but I don't have the resources
to do them. 

> Perhaps I was spoilt with Minion in previous projects.. personally I would
> have loved to have simply adopted it.. 

I wasn't super fond of Minion when it was first mentioned, although recently I
was re-reading the documentation and it is more full featured than I originally
thought. It probably would have been sufficient. In hindsight, we probably
should have gone that route. Considering the "sunk cost fallacy", perhaps we
should still go that way. Even in terms of sunk cost, we haven't actually
invested too much in RabbitMQ yet.

(Regarding RabbitMQ, I like how we can use a standard messaging protocol, which
gives us language agnosticism - it would be nice to use more task-appropriate
languages rather than just using Perl for everything. I like how it's scalable,
although few Koha instances need to scale. These days, you can use a managed
RabbitMQ via AWS, so you don't even have to maintain that infrastructure, if
you have the budget.)

> I thought the biggest thing against
> it was that it recommends the use or Mojo::Pg as the backend and we use
> MySQL.... 

I think that was one of the "cons", but when it was first mentioned I also
don't think that Koha had really adopted Mojolicious as fully as it has now. I
think there was still a fair bit of skepticism about Mojolicious at that point
in time.

> what's the difference in requiring people to install RabbitMQ vs
> requiring them to install Postgres?... 
> there is also a well maintained MySQL
> backend that implements a clever form of PubSub to get around the fact that
> MySQL doesn't do pubsub natively.
> 

I think it's a bit absurd to require 2 different relational databases on the
face of it, but that's a fair point from a technical perspective. I've noticed
the MySQL backend as well and it seems reasonable enough at a glance.

> As for a simpler scheduler.. why not just keep a list of tasks and execution
> times akin to cron in a table and query it every minute.. the exact same way
> cron does... and leave FIFO tasks to the queue implementation we've now got.

Many years ago, Galen and I talked about keeping a list of tasks and execution
times in a table and having a cronjob run every minute to examine the times and
run the tasks, but no one got around to doing it. Personally, I'm more
ambitious than that now and have had use cases that required scheduling
granularity of less than 1/minute. However, my ambitions certainly outstrip the
time/energy that I have available. Even though I have the technical ability, I
don't have the logistical resources to get it done in a timely way. 

Now I'm just being pedantic, but the simple scheduler proposal is different to
cron. If I recall correctly, cron keeps all the tasks in memory, and it only
does I/O when it detects a change to a crontab. But polling a database every
1/minute isn't a drama. It's a scheduler that wouldn't work for some of my use
cases, but it would meet the majority of Koha's needs, so it's probably the way
to go.

-- 
You are receiving this mail because:
You are watching all bug changes.