<div dir="ltr"><div><div class="gmail_quote"><div dir="ltr">On Thu, 23 Feb 2017 at 00:51 David Cook <<a href="mailto:dcook@prosentient.com.au">dcook@prosentient.com.au</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="blue" vlink="purple" lang="EN-AU"><div class="m_473387160322627662WordSection1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"></span><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I’m planning to post the code for what I have already in early March.</span></p></div></div></blockquote><div><br></div><div>Any news here? <br></div><div>We really need to remove the way our background jobs are implemented to make them work under Plack.<br></div><div>I'd like to avoid duplication of work...<br><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="blue" vlink="purple" lang="EN-AU"><div class="m_473387160322627662WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u><u></u></span></p></div></div><div link="blue" vlink="purple" lang="EN-AU"><div class="m_473387160322627662WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> David Cook</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Systems Librarian<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Prosentient Systems<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">72/330 Wattle St<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Ultimo, NSW 2007<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Australia<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Office: <a href="tel:02%2092%2012%2008%2099" value="+33292120899" target="_blank">02 9212 0899</a><u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Direct: <a href="tel:02%2080%2005%2005%2095" value="+33280050595" target="_blank">02 8005 0595</a><u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p></div></div><div link="blue" vlink="purple" lang="EN-AU"><div class="m_473387160322627662WordSection1"><div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"><div><div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" lang="EN-US">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" lang="EN-US"> Tomas Cohen Arazi [mailto:<a href="mailto:tomascohen@gmail.com" target="_blank">tomascohen@gmail.com</a>] <br><b>Sent:</b> Thursday, 23 February 2017 2:16 PM<br><b>To:</b> David Cook <<a href="mailto:dcook@prosentient.com.au" target="_blank">dcook@prosentient.com.au</a>>; Tajoli Zeno <<a href="mailto:z.tajoli@cineca.it" target="_blank">z.tajoli@cineca.it</a>>; <a href="mailto:koha-devel@lists.koha-community.org" target="_blank">koha-devel@lists.koha-community.org</a></span></p></div></div></div></div></div><div link="blue" vlink="purple" lang="EN-AU"><div class="m_473387160322627662WordSection1"><div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"><div><div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" lang="EN-US"><br><b>Subject:</b> Re: [Koha-devel] Task schedulers and message queues for Koha<u></u><u></u></span></p></div></div></div></div></div><div link="blue" vlink="purple" lang="EN-AU"><div class="m_473387160322627662WordSection1"><div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"><p class="MsoNormal"><u></u> <u></u></p><p>Share it :-)<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><div><div><p class="MsoNormal">El mié., 22 de feb. de 2017 9:57 PM, David Cook <<a href="mailto:dcook@prosentient.com.au" target="_blank">dcook@prosentient.com.au</a>> escribió:<u></u><u></u></p></div><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm"><p class="MsoNormal">Hi Zeno,<br><br>I have a number of concerns about Celery. One of those is that it would add<br>numerous external dependencies and complexity to Koha implementations.<br><br>Your suggestion of Celery + RabbitMQ + AnyEvent::RabbitMQ sounds ok,<br>although it would involve work too. While Celery clients exist for PHP and<br>Node.js, we'd need to create a Perl implementation of the Celery protocol<br>using AnyEvent::RabbitMQ (or Net::RabbitFoot). Not that I'm necessarily<br>opposed to that.<br><br>We'd also still need to write the tasks in Python (or use web hooks which<br>would have the overhead of HTTP plus you'd have to worry about your web<br>server being up). I'm not sure how keen the community at large is to support<br>more server-side languages. I like writing Python, so I don't mind porting<br>over my OAI-PMH code from Perl to Python. I've abandoned the HTTP::OAI<br>module anyway for a few reasons.<br><br>RabbitMQ is a pretty heavy duty product as well which comes with its own<br>requirements: <a href="https://www.rabbitmq.com/production-checklist.html" target="_blank">https://www.rabbitmq.com/production-checklist.html</a>. While we<br>currently help people with Apache, MySQL, Zebra, and ElasticSearch, we'd<br>also all need to become experts with RabbitMQ.<br><br>I've already put together a Perl-based scheduler using POE which forks its<br>own workers. And I've already put together a basic Perl-based message queue<br>which sends events to pre-existing workers (like Celery). Celery with<br>RabbitMQ is more mature and complex, but my Perl programs do the trick.<br><br>Looking at DSpace's OAI-PMH harvester, it works very much like my first<br>design. It's a Java scheduler which uses threads rather than child processes<br>to do its work.<br><br>Due to the lack of engagement overall, I think I'll probably just keep my<br>existing design, since it works and works quite well.<br><br>David Cook<br>Systems Librarian<br>Prosentient Systems<br>72/330 Wattle St<br>Ultimo, NSW 2007<br>Australia<br><br>Office: <a href="tel:02%2092%2012%2008%2099" value="+33292120899" target="_blank">02 9212 0899</a><br>Direct: <a href="tel:02%2080%2005%2005%2095" value="+33280050595" target="_blank">02 8005 0595</a><br><br><br>> -----Original Message-----<br>> From: Tajoli Zeno [mailto:<a href="mailto:z.tajoli@cineca.it" target="_blank">z.tajoli@cineca.it</a>]<br>> Sent: Wednesday, 22 February 2017 7:49 PM<br>> To: David Cook <<a href="mailto:dcook@prosentient.com.au" target="_blank">dcook@prosentient.com.au</a>>; <a href="mailto:koha-devel@lists.koha-" target="_blank">koha-devel@lists.koha-</a><br>> <a href="http://community.org" target="_blank">community.org</a><br>> Subject: Re: [Koha-devel] Task schedulers and message queues for Koha<br>><br>> Hi David and all,<br>><br>> Il 21/02/2017 23:29, David Cook ha scritto:<br>> >. Two, they wanted to<br>> > execute OAI-PMH requests every 2-3 seconds and cron has 1 minute as<br>> >its  finest granularity. Three, even if you setup a cronjob to run<br>> >every minute,  long running tasks could get duplicated (although you<br>> >could mitigate that  with locks which would be a pain). Plus, you want<br>> >to run tasks in parallel,  so you're going to want to use multiple<br>> >processes, which cron isn't really  set up to achieve.<br>><br>> Ok, if you need those features cron isn't enough.<br>> But why do you drop the option Celery +  RabbitMQ + AnyEvent::RabbitMQ<br>><br>> They have official debiano packages:<br>> <a href="https://packages.debian.org/jessie/python-celery" target="_blank">https://packages.debian.org/jessie/python-celery</a><br>> <a href="https://packages.debian.org/jessie/rabbitmq-server" target="_blank">https://packages.debian.org/jessie/rabbitmq-server</a><br>> <a href="https://packages.debian.org/jessie/libanyevent-rabbitmq-perl" target="_blank">https://packages.debian.org/jessie/libanyevent-rabbitmq-perl</a><br>><br>> We still use one of their dpendencies for similar tasks (libanyevent-perl<br>"event<br>> loop framework with multiple implementations").<br>><br>> Python is already present in our Debian/Ubuntu system, it is a prereq of<br>the<br>> distributions.<br>><br>> Redone a so complex stack in perl i think is very complex.<br>><br>> Bye<br>> Zeno Tajoli<br>><br>><br>><br>> --<br>> Zeno Tajoli<br>> /SVILUPPO PRODOTTI CINECA/ - Automazione Biblioteche<br>> Email: <a href="mailto:z.tajoli@cineca.it" target="_blank">z.tajoli@cineca.it</a> Fax: 051/6132198<br>> *CINECA* Consorzio Interuniversitario - Sede operativa di Segrate (MI)<br><br><br>_______________________________________________<br>Koha-devel mailing list<br><a href="mailto:Koha-devel@lists.koha-community.org" target="_blank">Koha-devel@lists.koha-community.org</a><br><a href="http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel" target="_blank">http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel</a><br>website : <a href="http://www.koha-community.org/" target="_blank">http://www.koha-community.org/</a><br>git : <a href="http://git.koha-community.org/" target="_blank">http://git.koha-community.org/</a><br>bugs : <a href="http://bugs.koha-community.org/" target="_blank">http://bugs.koha-community.org/</a><u></u><u></u></p></blockquote></div><div><p class="MsoNormal">-- <u></u><u></u></p></div><div><div><div><p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Helvetica",sans-serif;color:#757575">Tomás Cohen Arazi<u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Helvetica",sans-serif;color:#757575">Theke Solutions (<a href="http://theke.io/" target="_blank">https://theke.io</a>)<br></span><span style="font-size:9.5pt;font-family:"Segoe UI Symbol",sans-serif;color:#757575">✆</span><span style="font-size:9.5pt;font-family:"Helvetica",sans-serif;color:#757575"> <a href="tel:+54%209%20351%20351-3384" value="+5493513513384" target="_blank">+54 9351 3513384</a><br>GPG: B2F3C15F<u></u><u></u></span></p></div></div></div></div></div></div>_______________________________________________<br>
Koha-devel mailing list<br>
<a href="mailto:Koha-devel@lists.koha-community.org" target="_blank">Koha-devel@lists.koha-community.org</a><br>
<a href="http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel" rel="noreferrer" target="_blank">http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel</a><br>
website : <a href="http://www.koha-community.org/" rel="noreferrer" target="_blank">http://www.koha-community.org/</a><br>
git : <a href="http://git.koha-community.org/" rel="noreferrer" target="_blank">http://git.koha-community.org/</a><br>
bugs : <a href="http://bugs.koha-community.org/" rel="noreferrer" target="_blank">http://bugs.koha-community.org/</a></blockquote></div></div></div>