<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-AU link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal>Hi all,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>In 2016, I worked on a Koha task scheduler for downloading and importing records via OAI-PMH. I have code which works, but it’s lacking test coverage and I’m unsure that it will make it through QA and be accepted.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I recall Chris Cormack suggesting I look at Gearman (<a href="http://gearman.org/">http://gearman.org/</a>) instead, and it looks pretty good at a glance, although Andreas and I had talked about having more control over the workers than Gearman seems to offer. Plus, it added another dependency to Koha where people already struggle with dependencies.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Martin suggested that there were a lot of other implementations out there, but I haven’t really found much else that provides everything we want out of the box. The one I have found is Celery (<a href="http://www.celeryproject.org/">http://www.celeryproject.org/</a>), which looks like exactly what I want I think, but it requires Python for its server and workers, and it requires a message broker like RabbitMQ or Redis. Lots of extra dependencies.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Since my goal is to get our code into Koha, I really want to know what will work for people. Are people happy with a home grown solution? It’s not really that complicated. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>My current version is essentially a task scheduler which forks a worker on demand when it’s time to run a task (up to a configurable max of X tasks so you don’t kill your server). It lets you submit tasks, tell the scheduler to start/schedule them, and you can even tell in progress tasks to stop (by having the scheduler tell the worker to stop and the worker decides where in its task it checks for stop commands from the scheduler).<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>As I try to add test coverage and make this scheduler more palatable, I find myself thinking about the code more like Koha::Scheduler and Koha::Queue, and using the more scalable worker model used by others. The scheduler daemon would listen on a socket for tasks, it would create a Koha::Scheduler instance which would enqueue tasks to run once that task’s time was met or exceeded. Now depending on the architecture… you could have a separate daemon or the same daemon with a Koha::Queue instance. It would accept tasks/messages from the scheduler, and it would dequeue tasks to available workers - which are separate processes - have previously registered against particular queues. In this way, you can have a oaipmh-download queue, oaipmh-import queue, a email-report queue, etc.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I suspect that we could make use of Koha::Scheduler and Koha::Queue throughout much of Koha for doing background tasks. If we don’t want to reinvent the wheel with Koha::Queue, we could use something like RabbitMQ. But I think we need *something*.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I’m open to ideas. I already have the OAI-PMH download and OAI-PMH import handled. That’s the easy part. Any worker can do that. The hard part is figuring out how the Koha Community will take up a task scheduler. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='mso-fareast-language:EN-AU'>David Cook<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-AU'>Systems Librarian<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-AU'>Prosentient Systems<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-AU'>72/330 Wattle St<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-AU'>Ultimo, NSW 2007<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-AU'>Australia<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-AU'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-AU'>Office: 02 9212 0899<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-AU'>Direct: 02 8005 0595<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>