[Koha-devel] Task schedulers and message queues for Koha

Jonathan Druart jonathan.druart at bugs.koha-community.org
Tue Apr 25 18:21:04 CEST 2017


On Thu, 23 Feb 2017 at 00:51 David Cook <dcook at prosentient.com.au> wrote:

> I’m planning to post the code for what I have already in early March.
>

Any news here?
We really need to remove the way our background jobs are implemented to
make them work under Plack.
I'd like to avoid duplication of work...



>  David Cook
>
> Systems Librarian
>
> Prosentient Systems
>
> 72/330 Wattle St
>
> Ultimo, NSW 2007
>
> Australia
>
>
>
> Office: 02 9212 0899 <02%2092%2012%2008%2099>
>
> Direct: 02 8005 0595 <02%2080%2005%2005%2095>
>
>
>
> *From:* Tomas Cohen Arazi [mailto:tomascohen at gmail.com]
> *Sent:* Thursday, 23 February 2017 2:16 PM
> *To:* David Cook <dcook at prosentient.com.au>; Tajoli Zeno <
> z.tajoli at cineca.it>; koha-devel at lists.koha-community.org
>
>
> *Subject:* Re: [Koha-devel] Task schedulers and message queues for Koha
>
>
>
> Share it :-)
>
>
>
> El mié., 22 de feb. de 2017 9:57 PM, David Cook <dcook at prosentient.com.au>
> escribió:
>
> Hi Zeno,
>
> I have a number of concerns about Celery. One of those is that it would add
> numerous external dependencies and complexity to Koha implementations.
>
> Your suggestion of Celery + RabbitMQ + AnyEvent::RabbitMQ sounds ok,
> although it would involve work too. While Celery clients exist for PHP and
> Node.js, we'd need to create a Perl implementation of the Celery protocol
> using AnyEvent::RabbitMQ (or Net::RabbitFoot). Not that I'm necessarily
> opposed to that.
>
> We'd also still need to write the tasks in Python (or use web hooks which
> would have the overhead of HTTP plus you'd have to worry about your web
> server being up). I'm not sure how keen the community at large is to
> support
> more server-side languages. I like writing Python, so I don't mind porting
> over my OAI-PMH code from Perl to Python. I've abandoned the HTTP::OAI
> module anyway for a few reasons.
>
> RabbitMQ is a pretty heavy duty product as well which comes with its own
> requirements: https://www.rabbitmq.com/production-checklist.html. While we
> currently help people with Apache, MySQL, Zebra, and ElasticSearch, we'd
> also all need to become experts with RabbitMQ.
>
> I've already put together a Perl-based scheduler using POE which forks its
> own workers. And I've already put together a basic Perl-based message queue
> which sends events to pre-existing workers (like Celery). Celery with
> RabbitMQ is more mature and complex, but my Perl programs do the trick.
>
> Looking at DSpace's OAI-PMH harvester, it works very much like my first
> design. It's a Java scheduler which uses threads rather than child
> processes
> to do its work.
>
> Due to the lack of engagement overall, I think I'll probably just keep my
> existing design, since it works and works quite well.
>
> David Cook
> Systems Librarian
> Prosentient Systems
> 72/330 Wattle St
> Ultimo, NSW 2007
> Australia
>
> Office: 02 9212 0899 <02%2092%2012%2008%2099>
> Direct: 02 8005 0595 <02%2080%2005%2005%2095>
>
>
> > -----Original Message-----
> > From: Tajoli Zeno [mailto:z.tajoli at cineca.it]
> > Sent: Wednesday, 22 February 2017 7:49 PM
> > To: David Cook <dcook at prosentient.com.au>; koha-devel at lists.koha-
> > community.org
> > Subject: Re: [Koha-devel] Task schedulers and message queues for Koha
> >
> > Hi David and all,
> >
> > Il 21/02/2017 23:29, David Cook ha scritto:
> > >. Two, they wanted to
> > > execute OAI-PMH requests every 2-3 seconds and cron has 1 minute as
> > >its  finest granularity. Three, even if you setup a cronjob to run
> > >every minute,  long running tasks could get duplicated (although you
> > >could mitigate that  with locks which would be a pain). Plus, you want
> > >to run tasks in parallel,  so you're going to want to use multiple
> > >processes, which cron isn't really  set up to achieve.
> >
> > Ok, if you need those features cron isn't enough.
> > But why do you drop the option Celery +  RabbitMQ + AnyEvent::RabbitMQ
> >
> > They have official debiano packages:
> > https://packages.debian.org/jessie/python-celery
> > https://packages.debian.org/jessie/rabbitmq-server
> > https://packages.debian.org/jessie/libanyevent-rabbitmq-perl
> >
> > We still use one of their dpendencies for similar tasks (libanyevent-perl
> "event
> > loop framework with multiple implementations").
> >
> > Python is already present in our Debian/Ubuntu system, it is a prereq of
> the
> > distributions.
> >
> > Redone a so complex stack in perl i think is very complex.
> >
> > Bye
> > Zeno Tajoli
> >
> >
> >
> > --
> > Zeno Tajoli
> > /SVILUPPO PRODOTTI CINECA/ - Automazione Biblioteche
> > Email: z.tajoli at cineca.it Fax: 051/6132198
> > *CINECA* Consorzio Interuniversitario - Sede operativa di Segrate (MI)
>
>
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org
> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : http://www.koha-community.org/
> git : http://git.koha-community.org/
> bugs : http://bugs.koha-community.org/
>
> --
>
> Tomás Cohen Arazi
>
> Theke Solutions (https://theke.io <http://theke.io/>)
> ✆ +54 9351 3513384 <+54%209%20351%20351-3384>
> GPG: B2F3C15F
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org
> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : http://www.koha-community.org/
> git : http://git.koha-community.org/
> bugs : http://bugs.koha-community.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.koha-community.org/pipermail/koha-devel/attachments/20170425/e839eafa/attachment.html>


More information about the Koha-devel mailing list