February 2017 - Koha-devel - lists.koha-community.org

Task schedulers and message queues for Koha
by David Cook 04 May '17

04 May '17

Hi all, In 2016, I worked on a Koha task scheduler for downloading and importing records via OAI-PMH. I have code which works, but it's lacking test coverage and I'm unsure that it will make it through QA and be accepted. I recall Chris Cormack suggesting I look at Gearman (http://gearman.org/) instead, and it looks pretty good at a glance, although Andreas and I had talked about having more control over the workers than Gearman seems to offer. Plus, it added another dependency to Koha where people already struggle with dependencies. Martin suggested that there were a lot of other implementations out there, but I haven't really found much else that provides everything we want out of the box. The one I have found is Celery (http://www.celeryproject.org/) which looks like exactly what I want I think, but it requires Python for its server and workers, and it requires a message broker like RabbitMQ or Redis. Lots of extra dependencies. Since my goal is to get our code into Koha, I really want to know what will work for people. Are people happy with a home grown solution? It's not really that complicated. My current version is essentially a task scheduler which forks a worker on demand when it's time to run a task (up to a configurable max of X tasks so you don't kill your server). It lets you submit tasks, tell the scheduler to start/schedule them, and you can even tell in progress tasks to stop (by having the scheduler tell the worker to stop and the worker decides where in its task it checks for stop commands from the scheduler). As I try to add test coverage and make this scheduler more palatable, I find myself thinking about the code more like Koha::Scheduler and Koha::Queue, and using the more scalable worker model used by others. The scheduler daemon would listen on a socket for tasks, it would create a Koha::Scheduler instance which would enqueue tasks to run once that task's time was met or exceeded. Now depending on the architecture. you could have a separate daemon or the same daemon with a Koha::Queue instance. It would accept tasks/messages from the scheduler, and it would dequeue tasks to available workers - which are separate processes - have previously registered against particular queues. In this way, you can have a oaipmh-download queue, oaipmh-import queue, a email-report queue, etc. I suspect that we could make use of Koha::Scheduler and Koha::Queue throughout much of Koha for doing background tasks. If we don't want to reinvent the wheel with Koha::Queue, we could use something like RabbitMQ. But I think we need *something*. I'm open to ideas. I already have the OAI-PMH download and OAI-PMH import handled. That's the easy part. Any worker can do that. The hard part is figuring out how the Koha Community will take up a task scheduler. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595

4 8

Koha 3.22, Plack & EDS plugin
by Paul Poulain 27 Mar '17

27 Mar '17

Hello friends, We've deployed Koha 3.22, running Plack and the EDS plugin, and it appears that it does not work. We think it's because there are some .pl code that is out of plack scope, so it's not run. Is there anyone here who has some experience with koha / Plack / EDS ? (Frido will probably send a patch to Ebsco in the next days if there's no other ida) -- Paul Poulain, Associé-gérant / co-owner BibLibre, Services en logiciels libres pour les bibliothèques BibLibre, Open Source software and services for libraries

5 8

Interlibrary loans module
by Alex Sassmannshausen 24 Mar '17

24 Mar '17

Hello Kohites! Andrew and I have just finished a second major revision of our proposed interlibrary loans module for Koha. The code and bug can be found at [https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7317] We believe the code has now reached the maturity it requires for wider engagement, and, more importantly, it's now reached the maturity that it should be easy to get it up and running relatively easily in your development environments. Unfortunately we cannot deploy the module using sandboxes because backends are implemented as separate code projects. And we would *love* your comments thoughts and concerns about the module if you are interested in ILL! The final comment on the bug contains basic documentation for the Module as a whole, and also some installation instructions. But for reference, I've attached the same document to this email too. Below you will find some further high-level background and a roadmap of what we would like to achieve. For now, if you're interested in this module: • have a look at the bug, and get involved in the discussion! • try to set up the module in your dev environment. It should work easily on dev boxes and dev installs that track master. • let us know if you face any issues! • start a conversation with us if you might be interested in creating a backend for your country's / organization's ILL workflows. Finally, if you don't have access to a development environment, but you would be interested in becoming involved in this project, get in touch and we might be able to provide you with access to a testing environment! Best regards, Alex Sassmannshausen PTFS Europe 1 High-level background ═══════════════════════ The ILLModule aims to provide a core framework against which different ILL workflows can be implemented within Koha. It achieves this by using 2 new tables as data store, and by using an extensible backend system to create 'connectors' to ILL providers. The data store consists of the illrequests and the illrequestattributes tables. The former is a traditional table that stores essential values associated with an ILL request. The later is a key/value store, linked to a row in the former. This store can be used to store arbitrary data provided by a backend. The backends implement highly customizable workflows for several core steps in the ILL management process. At the same time, each backend can extend the core steps (called defined as the `core_status_graph` in Koha/Illrequest.pm) with their own additional steps (defined as the `status_graph` in a backend's Base.pm). Each of these steps, both core and extensions, in turn can define any number of 'stages' required to complete each individual step. Each step has access to a template include file, which can dispatch on 'stage'. This is mirrored by each step having a sub in a backend's Base.pm, which once again can dispatch on 'stage'. The subs in Base.pm have access to the full data store provided by the ILLModule; similarly the template includes have full access to Koha template features, including access to custom JS blocks through which, for instance, external APIs can be called. The main aim of this Koha module was to provide a core that is comprehensive enough to store core data to only have to implement ILL once in Koha, whilst being extensible enough so that virtually any ILL workflow can be implemented against this core. We believe we have achieved this. I'd be very interested to hear from you if you believe you have a workflow that cannot be captured by this (obviously, third party tools that do not provide API access will be virtually impossible to seamlessly integrate into Koha). 2 Roadmap ═════════ The roadmap starts from the current release of code on the bugzilla issue. • Publication of mature beta level code (21 February 2017) ⁃ public testing ⁃ public discussion ⁃ dogfooding • Augment core functionality (~ June 2017) ⁃ add advanced configuration options for the core module ⁃ add a standard system for configuring backends ⁃ allow for an 'unmediated workflow' ⁃ finalise 'generic requests' workflow • Meet Koha QA standards (~ July 2017) • Add unit tests to core ILLModule • Ensure UI respects Koha standards • Ensure Core code passes qa scripts • Addition of Documentation ⁃ contributing a chapter to the Koha manual ⁃ ensure all pod information is correct and complete ⁃ ('writing your own backend' tutorial?) • Provide consistent error handling ⁃ standard means through which a backend can 'throw' an error. ⁃ replace uses of die with this standard route • Integration into Koha core in Koha 17.11

4 6

Koha General IRC Meeting Reminder - 1 MArch 2017, 20 UTC
by Nick Clemens 28 Feb '17

28 Feb '17

Hello All, Just a reminder that tomorrow at 20 UTC is the monthly general IRC meeting for koha. Local time conversion here: http://www.timeanddate.com/worldclock/fixedtime.html?msg=Koha+IRC+General+M… The agenda is on the wiki here https://wiki.koha-community.org/wiki/General_IRC_meeting_1_March_2017 Please add anything you wish to discuss. See you soon! -Nick

1 0

Removal of api/swagger/definitions.t
by Tomas Cohen Arazi 28 Feb '17

28 Feb '17

Hi, I propose removing t/db_dependent/api/swagger/definitions.t for a simple reason: it is not obvious that our endpoints should mimick our object models. For instance, if some fields should not be exposed on the API, then they should not be tested for existence. Also, think of fields that have names that are kept for historical reasons and should better be changed on the published api. Maybe I went too far explaining it, but I think it should be removed. Regards -- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F

1 0

Hea v2
by Jonathan Druart 28 Feb '17

28 Feb '17

Hi developers, I have submitted a version 2 of Hea on bug 18066. There are changes to the server code (hea-ws and hea-app) and Koha. I will need help for testing. There are screenshots on bug 18066 comment 2, and a test plan on comment 7. It would be great to get feedbacks and signoff quickly :) Cheers, Jonathan https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18066

1 1

Help requested - new Koha installer and onboarding tool
by Jonathan Druart 25 Feb '17

25 Feb '17

Hello, As some of you may know, a lot of works have been made by Alex, Owen and <alexbuckley(a)catalyst.net.nz>Mark to provide a new Koha installer. It really looks awesome and it would be great to get it in quickly. An onboarding tool feature is added at the end of the installation process to create some mandatory data, like library, patron category, superlibrarian, etc. Everything is almost done and has been fully tested. However a big bug has been found during the QA review: some sample data files have been moved from a directory to another and the translated files have not been taken into account. Which leads to issues if you try to install Koha using a translated languages. Bug 17855 (onboarding tool) and bug 17942 (web installer style improvement) are ready (fully tested). But we need bug 18039 (move the installer files for translated languages) first. It is not easy to test everything for just one guy, so we would need one person for each language: de-DE, es-ES, fr-FR, fr-CA, , it-IT, nb-NO, pl-PL, ru-RU and uk-UA The goal is to: 1. apply 18039, then 17855 and finally 17942 (things are not clear on the bug reports, so please use my remote branch [1]) 2. Translate the template for your language 3. Try the new web installer process, using your language 4. Get amazed about how beautiful it is 5. Switch to master, update the templates 6. Select all the sample files, screenshots of the file list, DB dump 7. Switch to the new branch 8. Select all the sample files, screenshots of the file list, DB dump 9. Compare all what you can compare and make sure the behaviours before and after the patches are the same. Let me know if you have any questions. If you are using ru-RU and uk-UA, be around! These 2 directories have not been updated for ages and I am wondering if someone is using them. Cheers, Jonathan [1] https://github.com/joubu/Koha/tree/new_web_installer

3 2

Hackfest in Marseille, registration open !
by Paul Poulain 23 Feb '17

23 Feb '17

Hello, Registrations for the hackfest are open ! Drop me an email and I'll register you and add you to the usual google document ! Welcome already to the 5 ByWaterians and to Claire G. that have announced they'll be here ;-) (Magnus, we will miss you :( ) -- Paul Poulain, Associé-gérant / co-owner BibLibre, Services en logiciels libres pour les bibliothèques BibLibre, Open Source software and services for libraries

1 0

Summary: Koha Devel Meeting, 22 Februrary 2017
by Nick Clemens 22 Feb '17

22 Feb '17

Hi All, We had a productive dev meeting today, just wanted to highlight some of our discussions for the general list readers. If this seems useful I will try to send these updates after each dev meeting: The Onboarding tool <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17855> still needs more testing (see Jonathan's email here: http://lists.koha-community.org/pipermail/koha-devel/2017-February/043481.h…) This tool will make it easier for those setting up a new instance to enable necessary settings and configuration and generally make Koha setup more friendly. It was proposed as a possibility to roll the tool out in English first, and add other languages as they are tested. Ideally we would have as much testing as possible now. We will discuss and vote on how to move ahead at the next Developers meeting (8 March 2017, 19UTC) The ru-RU and uk-UA languages were discussed for possible removal as there has been no maintenance, if you are using these or interested in helping maintain (or have strong opinion) please let us know. Testing and QA teams can always use more help so let us know if you are interested. Other interesting notes: ElasticSearch dependencies are being packaged by Mirko Jonathan continues work on refactoring and moving things to the Koha namespace (see here <https://bugs.koha-community.org/bugzilla3/showdependencygraph.cgi?id=16846> for Koha::Patron status) OverDrive integration has been pushed to master The next meeting will be 8 March 2017, 19UTC <https://wiki.koha-community.org/wiki/Development_IRC_meeting_8_March_2017>, I hope to see you there.

1 0

Koha 16.11.04 released
by Katrin 22 Feb '17

22 Feb '17

The Koha community is proud to announce the release of Koha 16.11.04. This is a maintenance release of 16.11 and contains 3 enhancements and 41 bugfixes. The full release notes are available at https://koha-community.org/koha-16-11-04-release Packages will be available soon.

2 1