[Koha-devel] Task schedulers and message queues for Koha

[David Cook]

Which one, Tomas?

 

I’m planning to post the code for what I have already in early March.

 

David Cook

Systems Librarian

Prosentient Systems

72/330 Wattle St

Ultimo, NSW 2007

Australia

 

Office: 02 9212 0899

Direct: 02 8005 0595

 

From: Tomas Cohen Arazi [mailto:tomascohen at gmail.com] 
Sent: Thursday, 23 February 2017 2:16 PM
To: David Cook <dcook at prosentient.com.au>; Tajoli Zeno <z.tajoli at cineca.it>; koha-devel at lists.koha-community.org
Subject: Re: [Koha-devel] Task schedulers and message queues for Koha

 

Share it :-)

 

El mié., 22 de feb. de 2017 9:57 PM, David Cook <dcook at prosentient.com.au <mailto:dcook at prosentient.com.au> > escribió:

Hi Zeno,

I have a number of concerns about Celery. One of those is that it would add
numerous external dependencies and complexity to Koha implementations.

Your suggestion of Celery + RabbitMQ + AnyEvent::RabbitMQ sounds ok,
although it would involve work too. While Celery clients exist for PHP and
Node.js, we'd need to create a Perl implementation of the Celery protocol
using AnyEvent::RabbitMQ (or Net::RabbitFoot). Not that I'm necessarily
opposed to that.

We'd also still need to write the tasks in Python (or use web hooks which
would have the overhead of HTTP plus you'd have to worry about your web
server being up). I'm not sure how keen the community at large is to support
more server-side languages. I like writing Python, so I don't mind porting
over my OAI-PMH code from Perl to Python. I've abandoned the HTTP::OAI
module anyway for a few reasons.

RabbitMQ is a pretty heavy duty product as well which comes with its own
requirements: https://www.rabbitmq.com/production-checklist.html. While we
currently help people with Apache, MySQL, Zebra, and ElasticSearch, we'd
also all need to become experts with RabbitMQ.

I've already put together a Perl-based scheduler using POE which forks its
own workers. And I've already put together a basic Perl-based message queue
which sends events to pre-existing workers (like Celery). Celery with
RabbitMQ is more mature and complex, but my Perl programs do the trick.

Looking at DSpace's OAI-PMH harvester, it works very much like my first
design. It's a Java scheduler which uses threads rather than child processes
to do its work.

Due to the lack of engagement overall, I think I'll probably just keep my
existing design, since it works and works quite well.

David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia

Office: 02 9212 0899
Direct: 02 8005 0595


> -----Original Message-----
> From: Tajoli Zeno [mailto:z.tajoli at cineca.it <mailto:z.tajoli at cineca.it> ]
> Sent: Wednesday, 22 February 2017 7:49 PM
> To: David Cook <dcook at prosentient.com.au <mailto:dcook at prosentient.com.au> >; koha-devel at lists.koha- <mailto:koha-devel at lists.koha-> 
> community.org <http://community.org> 
> Subject: Re: [Koha-devel] Task schedulers and message queues for Koha
>
> Hi David and all,
>
> Il 21/02/2017 23:29, David Cook ha scritto:
> >. Two, they wanted to
> > execute OAI-PMH requests every 2-3 seconds and cron has 1 minute as
> >its  finest granularity. Three, even if you setup a cronjob to run
> >every minute,  long running tasks could get duplicated (although you
> >could mitigate that  with locks which would be a pain). Plus, you want
> >to run tasks in parallel,  so you're going to want to use multiple
> >processes, which cron isn't really  set up to achieve.
>
> Ok, if you need those features cron isn't enough.
> But why do you drop the option Celery +  RabbitMQ + AnyEvent::RabbitMQ
>
> They have official debiano packages:
> https://packages.debian.org/jessie/python-celery
> https://packages.debian.org/jessie/rabbitmq-server
> https://packages.debian.org/jessie/libanyevent-rabbitmq-perl
>
> We still use one of their dpendencies for similar tasks (libanyevent-perl
"event
> loop framework with multiple implementations").
>
> Python is already present in our Debian/Ubuntu system, it is a prereq of
the
> distributions.
>
> Redone a so complex stack in perl i think is very complex.
>
> Bye
> Zeno Tajoli
>
>
>
> --
> Zeno Tajoli
> /SVILUPPO PRODOTTI CINECA/ - Automazione Biblioteche
> Email: z.tajoli at cineca.it <mailto:z.tajoli at cineca.it>  Fax: 051/6132198
> *CINECA* Consorzio Interuniversitario - Sede operativa di Segrate (MI)


_______________________________________________
Koha-devel mailing list
Koha-devel at lists.koha-community.org <mailto:Koha-devel at lists.koha-community.org> 
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

-- 

Tomás Cohen Arazi

Theke Solutions (https://theke.io <http://theke.io/> )
✆ +54 9351 3513384
GPG: B2F3C15F

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.koha-community.org/pipermail/koha-devel/attachments/20170223/30040369/attachment.html>