-----Original Message----- From: Tajoli Zeno [mailto:z.tajoli@cineca.it] Sent: Wednesday, 22 February 2017 7:49 PM To: David Cook <dcook@prosentient.com.au>; koha-devel@lists.koha- community.org Subject: Re: [Koha-devel] Task schedulers and message queues for Koha
Hi David and all,
Il 21/02/2017 23:29, David Cook ha scritto:
. Two, they wanted to execute OAI-PMH requests every 2-3 seconds and cron has 1 minute as its finest granularity. Three, even if you setup a cronjob to run every minute, long running tasks could get duplicated (although you could mitigate that with locks which would be a pain). Plus, you want to run tasks in parallel, so you're going to want to use multiple processes, which cron isn't really set up to achieve.
Ok, if you need those features cron isn't enough. But why do you drop the option Celery + RabbitMQ + AnyEvent::RabbitMQ
They have official debiano packages: https://packages.debian.org/jessie/python-celery https://packages.debian.org/jessie/rabbitmq-server https://packages.debian.org/jessie/libanyevent-rabbitmq-perl
We still use one of their dpendencies for similar tasks (libanyevent-perl "event loop framework with multiple implementations").
Python is already present in our Debian/Ubuntu system, it is a prereq of
Hi Zeno, I have a number of concerns about Celery. One of those is that it would add numerous external dependencies and complexity to Koha implementations. Your suggestion of Celery + RabbitMQ + AnyEvent::RabbitMQ sounds ok, although it would involve work too. While Celery clients exist for PHP and Node.js, we'd need to create a Perl implementation of the Celery protocol using AnyEvent::RabbitMQ (or Net::RabbitFoot). Not that I'm necessarily opposed to that. We'd also still need to write the tasks in Python (or use web hooks which would have the overhead of HTTP plus you'd have to worry about your web server being up). I'm not sure how keen the community at large is to support more server-side languages. I like writing Python, so I don't mind porting over my OAI-PMH code from Perl to Python. I've abandoned the HTTP::OAI module anyway for a few reasons. RabbitMQ is a pretty heavy duty product as well which comes with its own requirements: https://www.rabbitmq.com/production-checklist.html. While we currently help people with Apache, MySQL, Zebra, and ElasticSearch, we'd also all need to become experts with RabbitMQ. I've already put together a Perl-based scheduler using POE which forks its own workers. And I've already put together a basic Perl-based message queue which sends events to pre-existing workers (like Celery). Celery with RabbitMQ is more mature and complex, but my Perl programs do the trick. Looking at DSpace's OAI-PMH harvester, it works very much like my first design. It's a Java scheduler which uses threads rather than child processes to do its work. Due to the lack of engagement overall, I think I'll probably just keep my existing design, since it works and works quite well. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595 the
distributions.
Redone a so complex stack in perl i think is very complex.
Bye Zeno Tajoli
-- Zeno Tajoli /SVILUPPO PRODOTTI CINECA/ - Automazione Biblioteche Email: z.tajoli@cineca.it Fax: 051/6132198 *CINECA* Consorzio Interuniversitario - Sede operativa di Segrate (MI)