<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Symbol";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-AU link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><a name="_MailEndCompose"><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Hi Jonathan,<o:p></o:p></span></a></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>I have scrapped the work that I was doing on a generic task scheduler. Instead, I’ve developed a daemon which just works for OAI-PMH harvesting. Feel free to go ahead with whatever you’re planning, and I’m happy to contribute ideas. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>I was concerned that I was being overly ambitious, and that the project would never be accepted into Koha. A third-party message queue like RabbitMQ would add another dependency to Koha, which would further complicate installations and maintainance, although I think it might still be the best way forward. Or using something like ZeroMQ to set up our own using some established work. I had written my own message queue in Perl, which was fairly easy to do, so that’s always an alternative. For the task scheduler, I used POE for the event framework and used timers to schedule tasks. Initially I didn’t use a message queue, but I think using one would be more optimal. You fire up some workers, register them with the message queue, and then they consume messages as the message queue assigns them. The task scheduler would then just be used for initially queuing the messages into the queue for the workers to consume.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>With the OAI-PMH daemon, which I’d like to post ASAP to 10662, I’m still using POE for the event framework, but I’m using POE::Component::JobQueue to handle the queue. I have a queue for downloading and a queue for importing. Each queue has X workers which run in parallel. At the moment, I’m forking the workers, since it was the easiest thing to do, but it is a little bit heavy. Not in terms of the overhead of forking, which is fairly non-existent really, but since you’re getting a copy of the harvester for each forked worker, the resources seem to add up a bit. At the moment, my design is for a single Koha system, or one with a lot of resources. Anyway, so the Koha web interface connects to the OAI-PMH harvester daemon using a UNIX socket. In koha-conf.xml, I have a line pointing to a configuration file, and in there is a socket address. It uses a super simple protocol serialised in JSON with null terminated lines to submit/list/start/stop/delete jobs in the harvester. The harvester downloads records to the file system and adds a pointer to the database, and then the importer job queue assigns a database entry to each of its workers and imports the records into Koha. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>I was thinking that even with a task scheduler and message queue, I’d probably still implement the OAI-PMH harvester as I have. Maybe I could replace the UNIX socket connection with the message queue, so the harvester consumes messages from the queue rather than the client, but it’s a bit academic. The harvester needs to have direct control over its workers rather than the queue sending messages to the workers, so that it can control the jobs directly. I’m not a huge fan of how the Python-based Celery scheduler manages cancelled jobs, although I found Celery to be a neat piece of work. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Anyway, long story short, no real news. I’ve abandoned making a generic task scheduler and message queue, and just made my to-purpose daemon which implements its own internal queue management for the sake of simplicity and efficacy. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>David Cook<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Systems Librarian<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Prosentient Systems<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>72/330 Wattle St<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Ultimo, NSW 2007<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Australia<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Office: 02 9212 0899<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Direct: 02 8005 0595<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'> koha-devel-bounces@lists.koha-community.org [mailto:koha-devel-bounces@lists.koha-community.org] <b>On Behalf Of </b>Jonathan Druart<br><b>Sent:</b> Wednesday, 26 April 2017 2:21 AM<br><b>To:</b> koha-devel@lists.koha-community.org<br><b>Subject:</b> Re: [Koha-devel] Task schedulers and message queues for Koha<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><div><div><p class=MsoNormal>On Thu, 23 Feb 2017 at 00:51 David Cook <<a href="mailto:dcook@prosentient.com.au">dcook@prosentient.com.au</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>I’m planning to post the code for what I have already in early March.</span><o:p></o:p></p></div></div></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Any news here? <o:p></o:p></p></div><div><p class=MsoNormal>We really need to remove the way our background jobs are implemented to make them work under Plack.<o:p></o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'>I'd like to avoid duplication of work...<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> David Cook</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Systems Librarian</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Prosentient Systems</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>72/330 Wattle St</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Ultimo, NSW 2007</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Australia</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Office: <a href="tel:02%2092%2012%2008%2099" target="_blank">02 9212 0899</a></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Direct: <a href="tel:02%2080%2005%2005%2095" target="_blank">02 8005 0595</a></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> </span><o:p></o:p></p></div></div><div><div><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'> Tomas Cohen Arazi [mailto:<a href="mailto:tomascohen@gmail.com" target="_blank">tomascohen@gmail.com</a>] <br><b>Sent:</b> Thursday, 23 February 2017 2:16 PM<br><b>To:</b> David Cook <<a href="mailto:dcook@prosentient.com.au" target="_blank">dcook@prosentient.com.au</a>>; Tajoli Zeno <<a href="mailto:z.tajoli@cineca.it" target="_blank">z.tajoli@cineca.it</a>>; <a href="mailto:koha-devel@lists.koha-community.org" target="_blank">koha-devel@lists.koha-community.org</a></span><o:p></o:p></p></div></div></div></div></div><div><div><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'><br><b>Subject:</b> Re: [Koha-devel] Task schedulers and message queues for Koha</span><o:p></o:p></p></div></div></div></div></div><div><div><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p>Share it :-)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>El mié., 22 de feb. de 2017 9:57 PM, David Cook <<a href="mailto:dcook@prosentient.com.au" target="_blank">dcook@prosentient.com.au</a>> escribió:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hi Zeno,<br><br>I have a number of concerns about Celery. One of those is that it would add<br>numerous external dependencies and complexity to Koha implementations.<br><br>Your suggestion of Celery + RabbitMQ + AnyEvent::RabbitMQ sounds ok,<br>although it would involve work too. While Celery clients exist for PHP and<br>Node.js, we'd need to create a Perl implementation of the Celery protocol<br>using AnyEvent::RabbitMQ (or Net::RabbitFoot). Not that I'm necessarily<br>opposed to that.<br><br>We'd also still need to write the tasks in Python (or use web hooks which<br>would have the overhead of HTTP plus you'd have to worry about your web<br>server being up). I'm not sure how keen the community at large is to support<br>more server-side languages. I like writing Python, so I don't mind porting<br>over my OAI-PMH code from Perl to Python. I've abandoned the HTTP::OAI<br>module anyway for a few reasons.<br><br>RabbitMQ is a pretty heavy duty product as well which comes with its own<br>requirements: <a href="https://www.rabbitmq.com/production-checklist.html" target="_blank">https://www.rabbitmq.com/production-checklist.html</a>. While we<br>currently help people with Apache, MySQL, Zebra, and ElasticSearch, we'd<br>also all need to become experts with RabbitMQ.<br><br>I've already put together a Perl-based scheduler using POE which forks its<br>own workers. And I've already put together a basic Perl-based message queue<br>which sends events to pre-existing workers (like Celery). Celery with<br>RabbitMQ is more mature and complex, but my Perl programs do the trick.<br><br>Looking at DSpace's OAI-PMH harvester, it works very much like my first<br>design. It's a Java scheduler which uses threads rather than child processes<br>to do its work.<br><br>Due to the lack of engagement overall, I think I'll probably just keep my<br>existing design, since it works and works quite well.<br><br>David Cook<br>Systems Librarian<br>Prosentient Systems<br>72/330 Wattle St<br>Ultimo, NSW 2007<br>Australia<br><br>Office: <a href="tel:02%2092%2012%2008%2099" target="_blank">02 9212 0899</a><br>Direct: <a href="tel:02%2080%2005%2005%2095" target="_blank">02 8005 0595</a><br><br><br>> -----Original Message-----<br>> From: Tajoli Zeno [mailto:<a href="mailto:z.tajoli@cineca.it" target="_blank">z.tajoli@cineca.it</a>]<br>> Sent: Wednesday, 22 February 2017 7:49 PM<br>> To: David Cook <<a href="mailto:dcook@prosentient.com.au" target="_blank">dcook@prosentient.com.au</a>>; <a href="mailto:koha-devel@lists.koha-" target="_blank">koha-devel@lists.koha-</a><br>> <a href="http://community.org" target="_blank">community.org</a><br>> Subject: Re: [Koha-devel] Task schedulers and message queues for Koha<br>><br>> Hi David and all,<br>><br>> Il 21/02/2017 23:29, David Cook ha scritto:<br>> >. Two, they wanted to<br>> > execute OAI-PMH requests every 2-3 seconds and cron has 1 minute as<br>> >its finest granularity. Three, even if you setup a cronjob to run<br>> >every minute, long running tasks could get duplicated (although you<br>> >could mitigate that with locks which would be a pain). Plus, you want<br>> >to run tasks in parallel, so you're going to want to use multiple<br>> >processes, which cron isn't really set up to achieve.<br>><br>> Ok, if you need those features cron isn't enough.<br>> But why do you drop the option Celery + RabbitMQ + AnyEvent::RabbitMQ<br>><br>> They have official debiano packages:<br>> <a href="https://packages.debian.org/jessie/python-celery" target="_blank">https://packages.debian.org/jessie/python-celery</a><br>> <a href="https://packages.debian.org/jessie/rabbitmq-server" target="_blank">https://packages.debian.org/jessie/rabbitmq-server</a><br>> <a href="https://packages.debian.org/jessie/libanyevent-rabbitmq-perl" target="_blank">https://packages.debian.org/jessie/libanyevent-rabbitmq-perl</a><br>><br>> We still use one of their dpendencies for similar tasks (libanyevent-perl<br>"event<br>> loop framework with multiple implementations").<br>><br>> Python is already present in our Debian/Ubuntu system, it is a prereq of<br>the<br>> distributions.<br>><br>> Redone a so complex stack in perl i think is very complex.<br>><br>> Bye<br>> Zeno Tajoli<br>><br>><br>><br>> --<br>> Zeno Tajoli<br>> /SVILUPPO PRODOTTI CINECA/ - Automazione Biblioteche<br>> Email: <a href="mailto:z.tajoli@cineca.it" target="_blank">z.tajoli@cineca.it</a> Fax: 051/6132198<br>> *CINECA* Consorzio Interuniversitario - Sede operativa di Segrate (MI)<br><br><br>_______________________________________________<br>Koha-devel mailing list<br><a href="mailto:Koha-devel@lists.koha-community.org" target="_blank">Koha-devel@lists.koha-community.org</a><br><a href="http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel" target="_blank">http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel</a><br>website : <a href="http://www.koha-community.org/" target="_blank">http://www.koha-community.org/</a><br>git : <a href="http://git.koha-community.org/" target="_blank">http://git.koha-community.org/</a><br>bugs : <a href="http://bugs.koha-community.org/" target="_blank">http://bugs.koha-community.org/</a><o:p></o:p></p></blockquote></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>-- <o:p></o:p></p></div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:9.5pt;font-family:"Helvetica",sans-serif;color:#757575'>Tomás Cohen Arazi</span><o:p></o:p></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:9.5pt;font-family:"Helvetica",sans-serif;color:#757575'>Theke Solutions (<a href="http://theke.io/" target="_blank">https://theke.io</a>)<br></span><span style='font-size:9.5pt;font-family:"Segoe UI Symbol",sans-serif;color:#757575'>✆</span><span style='font-size:9.5pt;font-family:"Helvetica",sans-serif;color:#757575'> <a href="tel:+54%209%20351%20351-3384" target="_blank">+54 9351 3513384</a><br>GPG: B2F3C15F</span><o:p></o:p></p></div></div></div></div></div></div><p class=MsoNormal>_______________________________________________<br>Koha-devel mailing list<br><a href="mailto:Koha-devel@lists.koha-community.org" target="_blank">Koha-devel@lists.koha-community.org</a><br><a href="http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel" target="_blank">http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel</a><br>website : <a href="http://www.koha-community.org/" target="_blank">http://www.koha-community.org/</a><br>git : <a href="http://git.koha-community.org/" target="_blank">http://git.koha-community.org/</a><br>bugs : <a href="http://bugs.koha-community.org/" target="_blank">http://bugs.koha-community.org/</a><o:p></o:p></p></blockquote></div></div></div></div></div></body></html>