October 2016 - Koha-devel - lists.koha-community.org

Koha 16.11 feature slush, feature freeze, string freeze, release dates
by David Cook 11 Oct '16

11 Oct '16

Hi all, Do we have any dates set for feature slush, feature freeze, string freeze, and release? I think the release is often done around November 26th or 28th? David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595

1 0

ICU Chains Library -- an easy way to make Koha better
by Barton Chittenden 10 Oct '16

10 Oct '16

After some conversations on the Koha mailing list, I re-factored some of the language specific ICU chains pages and created https://wiki.koha-community.org/wiki/ICU_Chains_Library Which documents language specific modifications to words-icu.xml (and, at least theoretically, phrases-icu.xml). If any devs know of modifications that they've made for other languages, I would *love* to see these added to that page -- five minutes of copy/paste on your part could easily save another member of the community days or weeks of effort. Thank you! --Barton

1 0

What's on in Koha devel #5 - Special hk16
by Jonathan Druart 08 Oct '16

08 Oct '16

Hello librarians and developers, I send a "What's on in Koha devel" email to koha-devel list each month. But this one is a bit special as I will focus on the different subjects you may heard of if you attend the hackfest in Marseille next week. It can be helpful for both librarians and developers! A lot of things are going on in the Koha ecosystem so it might be hard to follow all the discussions or to jump into one. So my attempted goal here is to compile the topics currently "à la mode" for Koha developers. In this email I will go a bit more into details than usual, to let you the opportunity to better understand the themes and to ask questions next week if you want to know more. I hope it will help you to choose the subject you want to be involved in during the hackfest. = Sandboxes = Sandboxes have been developed with the aim to help people without any technical skills to test patches submitted on the Koha bugtracker. You will find all the information you want on the dedicated wiki page https://wiki.koha-community.org/wiki/Sandboxes = Refactoring = During the last 3 (more?) years we have integrated DBIx::Class (a Perl ORM, Object-Relationnal Mapping) into Koha, for several (sometimes disputed) reasons. To take advantage of it, we are using it as much as possible through a home-made object module called Koha::Object. For the last year, a lot of legacy code has been rewritten and moved out of the C4 namespace to the new Koha namespace. For instance Koha::Virtualshelves replaced C4::VirtualShelves::Page (bug 14544), Koha::Libraries replaces C4::Branch (bug 15293). At the moment, the job focusses on moving the legacy authorised values from the C4::Koha module to Koha::AuthorisedValues (bug 15799) and the patrons/borrowers/users/members code from C4::Members to Koha::Patrons (bug 16846). For an overview of this refactoring work, please have a look at bug 15449 and its scary dependency graph: https://bugs.koha-community.org/bugzilla3/showdependencygraph.cgi?id=15449 Another refactoring work is about moving the biblioitems.marcxml content out of the biblioitems table. The idea is to create another table (biblio_metadata) to add the ability to store a record in different formats. The direct and major addition of this move would be to bring performance speed. See the comment 1 of bug 17196 for more information. = Speed improvements = During the last 2 releases, we have made a lot of speed improvements. That has been achieve with the consolidation and the stabilisation of our caching system and plack integration. If you are running a recent version of Koha (3.22, 16.05 or later) you should set them up correctly to fully enjoy the improvement. To be fair I must say that we have had to concentrate our efforts on these points because of our previous technical decisions (mainly related to DBIx::Class). Two main playgrounds: Plack and Memcached, that I will explain now. == Plack == I bet you already have heard of Plack, because Koha developers have been talking about it for ages. Basically it is just an interface between the web server (Apache, starman, etc.) and the perl application (Koha). When a user hits the a Koha url, a lot of files (Perl modules) are compiled. In CGI mode, this compilation step is done for every request. Using Plack they are compiled only once. The code will be put in RAM and other requests won't need to process all the modules again. It is an advantage since we are using DBIx::Class more and more and its schema is heavy to load. With Plack it is only loaded once. Koha is now considered as stable under Plack for the last versions of Koha, so you must use it. To know about known bugs, you can follow bug 7172 which gather Plack related known bugs together. At this time there is only one patch not yet pushed, bug 17392 (ping QAer!). Note that a timeout issue seems related to Plack and is not yet resolved, see: Bug 16714 - Unexpected logout with "IP address change" (with SessionRestrictionByIP set) == Caching system == The way we are caching "stuffs" in Koha has really been improved recently and will continue to! Important steps have been done in this area. We are using Memcached - a memory caching system - to store big bunch of data that we do not want to recalculate or retrieve from the database everytime. For instance: - the sysprefs. A lot of them are retrieved from the database for each request - the biblio frameworks - the holidays - ... a lot of other things could/will be cached! A big step has been done when we decided to introduce a 2-level caching mechanism (bug 16044): we have now a in-memory L1 cache (flush at every request) and a L2 cache (Memcached). To understand how this mechanisms are useful, imagine a simple scenario: launch a search which will return 20 results. For each result we need to know the MARC bibliographic framework of the record. Say they are all using the default frameworkcode. Without any caching mechanism, we retrieved from the database the whole structure, 20 times, once per result. With our new caching mechanism, the information for the default MARC bibliographic framework will be retrieved on the first request, then put in both L1 and L2 cache. The 19 other results will retrieve it from the L1 cache. If the same or another user does a search, the framework info will be retrieved from the L2 cache for the first result, and the L1 cache will be populated. The other results will retrieve it from the L1 cache. Not sure this is clear, but catch me if you need more details :) All of that to say that if you are a developer, there are 3 other bugs with ideas to improve again the caching mechanism. They are all of them in discussion: Bug 16140 - Only clear L1 cache when needed Bug 16079 - Retrieving system preferences from database via DBIx is not fast enough Bug 15341 - Performance - Retrieve all sysprefs at once One which would be nice to have is: Bug 17261 - Add memcached configuration info to about.pl It will permit to display the memcached configuration in the about page. = MySQL 5.7 compatibility = If you are interested in using Koha with MySQL 5.7 (the default version for the last Ubuntu 16.04), you should take a look at bug 17258 and its dependencies. Koha is not ready at all for the new default sql_mode configuration of this version of MySQL (STRICT_TRANS_TABLES). = Security issues = A lot of security issues (CSRF and XSS) have been fixed for the last 4 months, and almost all of these fixes have been backported to stable releases. They are all reported under the 2 following omnibus: Bug 17096 - [OMNIBUS] CSRF protections Bug 14568 - [OMNIBUS] XSS in Staff Client Only one known bug is waiting to be QAed (but 17365). = Elastic search = There are no big new features pushed since the first big push. But a few bug fixes and enhancement are waiting to be QAed. You can find them on the dependency graph of bug 12478. Note that there are 2 known bugs, without patches: Bug 16660 - Elasticsearch broken if OpacSuppression is activated Bug 17373 - Elasticsearch - Authority mappings are not defined for UNIMARC Are there some UNIMARC users around? :) = RESTful API = Tons of patches have been submitted on bugzilla, but only few got attention from signoffers. If you are interested in testing of them, search for "rest api". = ReactJS = React is a JS library aimed to easier DOM manipulations. This topic is a recurrent one and developers involved in this discussion will be present at the hackfest. They should organise a discussion to reach a consensus once and for all. If accepted, it would be good to define guidelines and write complete examples. = Koha plugin system = It seems that people are curious about our plugin system. It would be good to revive the discussion on new and more powerful plugin system. See the discussion on the koha-devel list at http://lists.koha-community.org/pipermail/koha-devel/2016-May/042673.html = Transactions & exceptions = It would be good to see some developers organise a brainstorming session on the different topics that Tomas raised on his email to koha-devel a few weeks ago. http://lists.koha-community.org/pipermail/koha-devel/2016-September/043032.… = Mana = Paul presented us Mana (https://lists.katipo.co.nz/pipermail/koha/2016-July/045739.html) at the beginning of July. Morgan finished her internship and will be there are the hackfest to show us what she developed. Testers and QAers would certainly be welcomed to test and review this feature. = Hea = Hea is a website (http://hea.koha-community.org) collecting usage statistics from different Koha installations around the world. This feature is not enabled by default but is very useful for developers and other people loving statistics (so librarians isn't it?). If it is not done yet, you should enable this feature to join the 496 libraries already registered! One enhancement is developed and waiting for testers: 14608 - HEA : add possibility of sharing usage statistics in Administration page and Web installer Hope to see you ready to fix, translate, test or QA patches (and drink beers when everything is done) this week! Cheers, Jonathan

1 0

Migrating from koha 2.2 to koha 16.05
by Rodrigo Santellan 06 Oct '16

06 Oct '16

Hi, I have a question about the locations and from where it get the data of the marcs. For what I understand after running the move_marc_to_biblioitems.pl two new columns are created, marc and marcxml. And it moves all the data from marc_subfield_table to those fields. That has happen alright, my problem now is that I see the marcxml record this fields: <datafield tag="900" ind1=" " ind2=" "> <subfield code="b">n</subfield> <subfield code="c">a</subfield> <subfield code="d">m</subfield> <subfield code="g">2</subfield> <subfield code="h">2</subfield> <subfield code="k">a</subfield> </datafield> <datafield tag="908" ind1=" " ind2=" "> <subfield code="1">090903</subfield> <subfield code="2">t</subfield> <subfield code="3">2007</subfield> <subfield code="5">XXK</subfield> <subfield code="6">ENG</subfield> <subfield code="8">d</subfield> <subfield code="a">a</subfield> <subfield code="b">g</subfield> <subfield code="c">r</subfield> <subfield code="d">n</subfield> <subfield code="f">0</subfield> <subfield code="g">0</subfield> <subfield code="h">1</subfield> <subfield code="j">0</subfield> <subfield code="z">1</subfield> </datafield> But on the interface I can't find the fields 900 or 908! Not in the MARCdetail.pl or the addbiblio.pl those fields are empty. For what I see It take the value from there but is not populating them. What I'm missing? Regards and thanks for the help!!

1 0

Jenkins
by Mark Tompsett 06 Oct '16

06 Oct '16

Greetings, http://jenkins.koha-community.org/ They are all offline. With Kyle Hall’s patching up the the Kitchen Sink plug-in with JD’s patch, I was hoping to see a current run with happiness. Any idea what is going on? GPML, Mark Tompsett

1 0

Sandboxes Signoff
by Fridolin SOMERS 06 Oct '16

06 Oct '16

Hie, I'm setting up our new sandboxes. Applying patches is ok but not signoff. In bugzilla the status turns to "signed off", the current patch is set obsolete but the new patch is not attached. Applying patch now calls misc/devel/update_dbix_class_files.pl, with may change some files in sources. I may be the cause of the problem. Maybe add git stash / git stash pop to sandbox_signoff.cron ? Best regards, -- Fridolin SOMERS Biblibre - Pôles support et système fridolin.somers(a)biblibre.com

1 0

How do we find consensus on interface changes?
by Owen Leonard 06 Oct '16

06 Oct '16

Do we need a procedure for getting consensus on Koha interface changes? I'm thinking about this issue in part because of a couple of interface changes I've been working on. One for Bug 16421: Move hold edit actions into dropdown menu https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16421 (screenshot: https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=51111) ...and another for Bug 11375: Patron permission names are untranslatable https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11375 (screenshot: https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=51847) When proposing a change like that should there be a more inclusive process than getting one sign-off and one QA approval? If so, how do we decide what kind of changes require wider approval? How do we solicit opinions from a wider audience? Or is the process working well as-is? -- Owen -- Web Developer Athens County Public Libraries http://www.myacpl.org

9 15

Koha packages: a directory for spooled data
by David Cook 06 Oct '16

06 Oct '16

Hi all, What would be a good directory to use for spooled data when using Koha packages? Ideally, I'd like to use /var/spool/koha/<koha-instance>/. However, only the root user is about to write to that directory. This isn't really a problem for source installs or dev installs, since we control our own permissions, but it'll be a problem for package installs. I suppose a person could use /tmp, but that's not really appropriate, since /tmp isn't necessarily persistent. You might have a /tmp with tmpfs, have a server reboot, and then lose your spooled data. The answer to me is to allow the koha-instance user to write to /var/spool/koha/<koha-instance> as well. However, I want to hear what other people think, and hear about any alternative directory ideas. Cheers, David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595

1 0

Koha 3.22/16.05 & Apache, Plack for a high load/large library
by Paul Poulain 04 Oct '16

04 Oct '16

Hello koha-devel, Is there someone using Apache, Plack, on a high load server (ie: for a large library) ? Which configuration are you using (OS version, how is Koha installed -package or not-, any tuning in the config files ?) Do you face any problem ? We (BibLibre) face many cases where we get failures ("internal server error". Seems there's something like Apache not being able to wait long enough for Plack to answer). -- Paul Poulain, Associé-gérant / co-owner BibLibre, Services en logiciels libres pour les bibliothèques BibLibre, Open Source software and services for libraries

5 4

Improving the API for record imports
by David Cook 03 Oct '16

03 Oct '16

Hi all, I want to improve the API(s) for importing (bibliographic) records. But I'm not 100% sure how to go about what criteria to use to gauge "improvement". Here are some ideas: * I think the REST API offers up one obvious addition and that's DELETE. It would be great to delete records using the API. I've found this mostly thanks to OAI-PMH where you get a notice when a record has been deleted upstream and needs to be deleted from Koha too. o The difficulty in deleting bibliographic records is items. If there are items, it blocks the delete. You could auto-delete items, but you can't if there are holds or if it's currently issued. At best, you could return a failed delete via the API, I think. * I keep thinking that "import profiles" could be a good way to go too. This might include a MARC Modification Template, it might be a XSLT, it might be a Koha::Filter. It would let you turns your incoming metadata from format A to format B. o This would also allow you to POST a minimum amount of data to the API. You'd have your metadata record and your import profile code. * The downside of this is the POST wouldn't detail the options for the import, and those imports would be more obfuscated. Changes to a profile could also impact subsequent imports unintentionally. o I'm not sure how batches would work with filtering. You might have 50/100 records filter correctly and import, but that leaves 50 which aren't imported. Do you undo the whole import, or do you fix your filter, and then re-import the first 50 alongside the first time import of the last 50? * The original metadata would be stored regardless of metadata format (e.g. marcxml, mods, rdf, dc, etc.) for historical purposes or to allow re-filtering in the event that a filter fails to correctly transform a record to valid MARCXML (or whatever internal metadata format Koha uses). * The import source and the type of import would be stored (ie http://oai-server.com and "oai-pmh", lx2.loc.gov and "z3950", <insert relevant source> and "oclc connexion", etc.). o I think this would be helpful for managing/processing a certain type of import, perhaps also in terms of matching records. o At the moment, with my work on #10662, I have a separate table "import_oai" which obviously is for records imported via oai-pmh only. I don't actually track the origin due to how I currently source the imports, but I could do it in the future. * Storing some sort of identifier would be good o OAI-PMH records have unique identifiers for the records o Z39.50. that's tougher since it's not a specific metadata format like the OAI-PMH format. So I don't know about this one. Anyway, these are just some ideas I've been having. They're not fully formed by any stretch of the imagination. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595

4 4