Hi all,
In 2016, I worked on a Koha task scheduler for downloading and importing
records via OAI-PMH. I have code which works, but it's lacking test coverage
and I'm unsure that it will make it through QA and be accepted.
I recall Chris Cormack suggesting I look at Gearman (http://gearman.org/)
instead, and it looks pretty good at a glance, although Andreas and I had
talked about having more control over the workers than Gearman seems to
offer. Plus, it added another dependency to Koha where people already
struggle with dependencies.
Martin suggested that there were a lot of other implementations out there,
but I haven't really found much else that provides everything we want out of
the box. The one I have found is Celery (http://www.celeryproject.org/)
which looks like exactly what I want I think, but it requires Python for its
server and workers, and it requires a message broker like RabbitMQ or Redis.
Lots of extra dependencies.
Since my goal is to get our code into Koha, I really want to know what will
work for people. Are people happy with a home grown solution? It's not
really that complicated.
My current version is essentially a task scheduler which forks a worker on
demand when it's time to run a task (up to a configurable max of X tasks so
you don't kill your server). It lets you submit tasks, tell the scheduler to
start/schedule them, and you can even tell in progress tasks to stop (by
having the scheduler tell the worker to stop and the worker decides where in
its task it checks for stop commands from the scheduler).
As I try to add test coverage and make this scheduler more palatable, I find
myself thinking about the code more like Koha::Scheduler and Koha::Queue,
and using the more scalable worker model used by others. The scheduler
daemon would listen on a socket for tasks, it would create a Koha::Scheduler
instance which would enqueue tasks to run once that task's time was met or
exceeded. Now depending on the architecture. you could have a separate
daemon or the same daemon with a Koha::Queue instance. It would accept
tasks/messages from the scheduler, and it would dequeue tasks to available
workers - which are separate processes - have previously registered against
particular queues. In this way, you can have a oaipmh-download queue,
oaipmh-import queue, a email-report queue, etc.
I suspect that we could make use of Koha::Scheduler and Koha::Queue
throughout much of Koha for doing background tasks. If we don't want to
reinvent the wheel with Koha::Queue, we could use something like RabbitMQ.
But I think we need *something*.
I'm open to ideas. I already have the OAI-PMH download and OAI-PMH import
handled. That's the easy part. Any worker can do that. The hard part is
figuring out how the Koha Community will take up a task scheduler.
David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia
Office: 02 9212 0899
Direct: 02 8005 0595
Hi All,
Just a reminder that our next general meeting is tomorrow May 3rd in IRC at
20 UTC
Link to local date and time conversion:
http://www.timeanddate.com/worldclock/fixedtime.html?msg=Koha+IRC+General+M…
The agenda is here:
https://wiki.koha-community.org/wiki/General_IRC_meeting_03_May_2017
This meeting will be elections for 17.11 - you can see candidates for roles
here:
https://wiki.koha-community.org/wiki/Roles_for_17.11
Feature slush is on Friday, so make some noise tomorrow for anything you
want to see moving along.
Don't forget, next Wednesday will be a special Bugs Only! Global Bug
Squashing Day to get things moving for outstanding bugs before the next
release.
If you want to get involved meetings and GBSD are good points of entry so
hope to see you all there!
-Nick
Fellow Koha users:
The Koha Community Newsletter for April 2017 is here:
Koha Community Newsletter: April 2017
<https://koha-community.org/koha-community-newsletter-april-2017/>
Many thanks to the folks who submitted articles and news to this month's
newsletter.
Please feel free to email me with any corrections or suggestions.
Thanks!
--
Chad Roseburg
Editor, Koha Community Newsletter
