- Koha-devel - lists.koha-community.org

Seeking comments on Bug 17498 - Expose system preference values to JavaScript in the OPAC
by Owen Leonard 26 Oct '16

26 Oct '16

I've been working on an update to the way the OPAC handles JavaScript, with the long-term goal of being able to consolidate scripts in order to take advantage of JS concatenation and minification as much as possible. Bug 17498 describes what I've been testing: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17498 I'd appreciate any comments anyone might have. If the idea sounds crazy someone should let me know. -- Owen -- Web Developer Athens County Public Libraries http://www.myacpl.org

2 1

Re: [Koha-devel] [Koha] F5 Attacks
by Marcel de Rooy 26 Oct '16

26 Oct '16

More something for the developers list? What Philippe here says, makes some sense to me. We could at least try to do something; what and how is another thing ;) ________________________________ Van: Koha <koha-bounces(a)lists.katipo.co.nz> namens Philippe Blouin <philippe.blouin(a)inlibro.com> Verzonden: woensdag 26 oktober 2016 14:52 Aan: Koha list Onderwerp: Re: [Koha] F5 Attacks I disagree. If Koha is offered out of the box, and we take time to fix security issues, then it's normal for users to expect "basic" attacks to be taken care of. More so, blocking IP is not a possibility if genuine users are involved using a station from within the library. I'm not saying you're wrong that it's mostly sysadmin work and not Koha, but it doesn't mean nothing can be done. From the apache's threads, I found nothing useful (mostly derisive comments). But we could at least talk about it. What about having a javascript preventing refresh on the page withing 5 sec of each other? Needs to be done in a way that the refresh doesn't restart the timer. What about having the OPAC search be code where the refresh will basically send nothing ? The checkbox are filled, the request is sent to the backend, but the frontend keeps nothing... I'm just smoking here, but I'm trying to induce some brainstorming in this interesting topic. Philippe Blouin, Responsable du développement informatique Tél. : (888) 604-2627 philippe.blouin(a)inLibro.com <mailto:philippe.blouin@inLibro.com> inLibro | pour esprit libre | www.inLibro.com<http://www.inLibro.com> <http://www.inLibro.com> On 10/26/2016 07:13 AM, Jonathan Druart wrote: > Hi, > I don't think this can/must be fixed on Koha side. > It's a sysadmin duty to take care of that. > I would take a look at fail2ban to parse the web server access logs. But > make sure not to block your X librarians using the same ip ;) > > On Wed, 26 Oct 2016 at 12:28 Pedro Amorim <pjamorim91(a)gmail.com> wrote: > >> I have tested this and the stress caused on the server is very severe. It >> seems that for every request, a new zebra process is created and the server >> will only respond when the last one is finished. This ofc will result in >> time outs and eventually a crash in the server. >> >> This is a major critical issue IMO, anyone who knows about this has the >> power to deny the service of any Koha online without using any additional >> hacking/attacking software. >> >> The Koha I'm working on right now - still in development - is accessed >> behind a proxy server, and I will attempt to solve the problem through >> that, by limiting the requests from the same origin with very little time >> between them. Still, even if I'm successful with this, the problem will >> still lie in Koha. >> >> Anyone with some sort of insight is very welcome. >> >> Pedro Amorim >> >> 2016-10-26 8:24 GMT+00:00 clint.deckard <clint.deckard(a)frontiers.co.nz>: >> >>> I have had this issue appear today. I have attempted to set up >> mod_evasive >>> for apache but it doesn't seem to have solved the problem. >>> I would really appreciate some advice. >>> Clint. >>> >>> >>> rfblanchard wrote: >>> >>>> Assume a basic opac search: >>>> http://..../cgi-bin/koha/opac-search.pl?q=dog&branch_group_l >>>> imit=branch%3A349 >>>> >>>> This would take about 10 seconds to return the first time. >>>> >>>> Assume the user refreshes the results using f5 and keep there finger >>>> there a >>>> moment to long (3s): >>>> This would kill my server for about 1 minute. >>>> >>>> Any attacker could easily make the server unresponsive indefinitely by >>>> simply holding f5 on an opac search. >>>> >>>> Any recommendations on how to deal with this problem? >>>> >>>> here is a sample from top: >>>> >>>> Tasks: 313 total, 3 running, 309 sleeping, 0 stopped, 1 zombie >>>> %Cpu(s): 93.7 us, 5.2 sy, 0.0 ni, 1.0 id, 0.2 wa, 0.0 hi, 0.0 si, >>>> 0.0 >>>> st >>>> KiB Mem: 16465036 total, 1532492 used, 14932544 free, 63180 buffers >>>> KiB Swap: 8526844 total, 0 used, 8526844 free. 505124 cached >>>> Mem >>>> >>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >>>> COMMAND >>>> 7027 peischo+ 20 0 416164 162924 12756 S 58.8 1.0 0:26.43 >>>> /usr/share/koha >>>> 7009 peischo+ 20 0 416800 163524 12756 S 56.5 1.0 0:33.77 >>>> /usr/share/koha >>>> 7444 peischo+ 20 0 129832 15216 5900 R 37.2 0.1 0:01.12 >>>> zebrasrv >>>> 7445 peischo+ 20 0 129832 15216 5900 R 35.6 0.1 0:01.07 >>>> zebrasrv >>>> 1151 mysql 20 0 886564 181096 10808 S 8.6 1.1 1:27.57 >> mysqld >>>> 7435 koha 20 0 25892 3272 2528 R 0.3 0.0 0:00.03 top >>>> 1 root 20 0 176144 5044 3096 S 0.0 0.0 0:01.43 >>>> systemd >>>> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 >>>> kthreadd >>>> >>>> >>>> >>>> -- >>>> View this message in context: http://koha.1045719.n5.nabble. >>>> com/F5-Attacks-tp5906098.html >>>> Sent from the Koha-general mailing list archive at Nabble.com. >>>> _______________________________________________ >>>> Koha mailing list http://koha-community.org >>>> Koha(a)lists.katipo.co.nz >>>> https://lists.katipo.co.nz/mailman/listinfo/koha >>>> >>> _______________________________________________ >>> Koha mailing list http://koha-community.org >>> Koha(a)lists.katipo.co.nz >>> https://lists.katipo.co.nz/mailman/listinfo/koha >>> >> _______________________________________________ >> Koha mailing list http://koha-community.org >> Koha(a)lists.katipo.co.nz >> https://lists.katipo.co.nz/mailman/listinfo/koha >> > _______________________________________________ > Koha mailing list http://koha-community.org > Koha(a)lists.katipo.co.nz > https://lists.katipo.co.nz/mailman/listinfo/koha _______________________________________________ Koha mailing list http://koha-community.org Koha(a)lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha

1 0

Re: [Koha-devel] What's on in Koha devel #5 - Special hk16
by Kyle Hall 24 Oct '16

24 Oct '16

Thanks for the overview Jonathan! I know I for one have the react discussion as a top priority for the jacket. Kyle Sent from my phone. Please excuse my brevity. On Oct 8, 2016 2:20 PM, "Jonathan Druart" < jonathan.druart(a)bugs.koha-community.org> wrote: Hello librarians and developers, I send a "What's on in Koha devel" email to koha-devel list each month. But this one is a bit special as I will focus on the different subjects you may heard of if you attend the hackfest in Marseille next week. It can be helpful for both librarians and developers! A lot of things are going on in the Koha ecosystem so it might be hard to follow all the discussions or to jump into one. So my attempted goal here is to compile the topics currently "à la mode" for Koha developers. In this email I will go a bit more into details than usual, to let you the opportunity to better understand the themes and to ask questions next week if you want to know more. I hope it will help you to choose the subject you want to be involved in during the hackfest. = Sandboxes = Sandboxes have been developed with the aim to help people without any technical skills to test patches submitted on the Koha bugtracker. You will find all the information you want on the dedicated wiki page https://wiki.koha-community.org/wiki/Sandboxes = Refactoring = During the last 3 (more?) years we have integrated DBIx::Class (a Perl ORM, Object-Relationnal Mapping) into Koha, for several (sometimes disputed) reasons. To take advantage of it, we are using it as much as possible through a home-made object module called Koha::Object. For the last year, a lot of legacy code has been rewritten and moved out of the C4 namespace to the new Koha namespace. For instance Koha::Virtualshelves replaced C4::VirtualShelves::Page (bug 14544), Koha::Libraries replaces C4::Branch (bug 15293). At the moment, the job focusses on moving the legacy authorised values from the C4::Koha module to Koha::AuthorisedValues (bug 15799) and the patrons/borrowers/users/members code from C4::Members to Koha::Patrons (bug 16846). For an overview of this refactoring work, please have a look at bug 15449 and its scary dependency graph: https://bugs.koha-community.org/bugzilla3/showdependencygraph.cgi?id=15449 Another refactoring work is about moving the biblioitems.marcxml content out of the biblioitems table. The idea is to create another table (biblio_metadata) to add the ability to store a record in different formats. The direct and major addition of this move would be to bring performance speed. See the comment 1 of bug 17196 for more information. = Speed improvements = During the last 2 releases, we have made a lot of speed improvements. That has been achieve with the consolidation and the stabilisation of our caching system and plack integration. If you are running a recent version of Koha (3.22, 16.05 or later) you should set them up correctly to fully enjoy the improvement. To be fair I must say that we have had to concentrate our efforts on these points because of our previous technical decisions (mainly related to DBIx::Class). Two main playgrounds: Plack and Memcached, that I will explain now. == Plack == I bet you already have heard of Plack, because Koha developers have been talking about it for ages. Basically it is just an interface between the web server (Apache, starman, etc.) and the perl application (Koha). When a user hits the a Koha url, a lot of files (Perl modules) are compiled. In CGI mode, this compilation step is done for every request. Using Plack they are compiled only once. The code will be put in RAM and other requests won't need to process all the modules again. It is an advantage since we are using DBIx::Class more and more and its schema is heavy to load. With Plack it is only loaded once. Koha is now considered as stable under Plack for the last versions of Koha, so you must use it. To know about known bugs, you can follow bug 7172 which gather Plack related known bugs together. At this time there is only one patch not yet pushed, bug 17392 (ping QAer!). Note that a timeout issue seems related to Plack and is not yet resolved, see: Bug 16714 - Unexpected logout with "IP address change" (with SessionRestrictionByIP set) == Caching system == The way we are caching "stuffs" in Koha has really been improved recently and will continue to! Important steps have been done in this area. We are using Memcached - a memory caching system - to store big bunch of data that we do not want to recalculate or retrieve from the database everytime. For instance: - the sysprefs. A lot of them are retrieved from the database for each request - the biblio frameworks - the holidays - ... a lot of other things could/will be cached! A big step has been done when we decided to introduce a 2-level caching mechanism (bug 16044): we have now a in-memory L1 cache (flush at every request) and a L2 cache (Memcached). To understand how this mechanisms are useful, imagine a simple scenario: launch a search which will return 20 results. For each result we need to know the MARC bibliographic framework of the record. Say they are all using the default frameworkcode. Without any caching mechanism, we retrieved from the database the whole structure, 20 times, once per result. With our new caching mechanism, the information for the default MARC bibliographic framework will be retrieved on the first request, then put in both L1 and L2 cache. The 19 other results will retrieve it from the L1 cache. If the same or another user does a search, the framework info will be retrieved from the L2 cache for the first result, and the L1 cache will be populated. The other results will retrieve it from the L1 cache. Not sure this is clear, but catch me if you need more details :) All of that to say that if you are a developer, there are 3 other bugs with ideas to improve again the caching mechanism. They are all of them in discussion: Bug 16140 - Only clear L1 cache when needed Bug 16079 - Retrieving system preferences from database via DBIx is not fast enough Bug 15341 - Performance - Retrieve all sysprefs at once One which would be nice to have is: Bug 17261 - Add memcached configuration info to about.pl It will permit to display the memcached configuration in the about page. = MySQL 5.7 compatibility = If you are interested in using Koha with MySQL 5.7 (the default version for the last Ubuntu 16.04), you should take a look at bug 17258 and its dependencies. Koha is not ready at all for the new default sql_mode configuration of this version of MySQL (STRICT_TRANS_TABLES). = Security issues = A lot of security issues (CSRF and XSS) have been fixed for the last 4 months, and almost all of these fixes have been backported to stable releases. They are all reported under the 2 following omnibus: Bug 17096 - [OMNIBUS] CSRF protections Bug 14568 - [OMNIBUS] XSS in Staff Client Only one known bug is waiting to be QAed (but 17365). = Elastic search = There are no big new features pushed since the first big push. But a few bug fixes and enhancement are waiting to be QAed. You can find them on the dependency graph of bug 12478. Note that there are 2 known bugs, without patches: Bug 16660 - Elasticsearch broken if OpacSuppression is activated Bug 17373 - Elasticsearch - Authority mappings are not defined for UNIMARC Are there some UNIMARC users around? :) = RESTful API = Tons of patches have been submitted on bugzilla, but only few got attention from signoffers. If you are interested in testing of them, search for "rest api". = ReactJS = React is a JS library aimed to easier DOM manipulations. This topic is a recurrent one and developers involved in this discussion will be present at the hackfest. They should organise a discussion to reach a consensus once and for all. If accepted, it would be good to define guidelines and write complete examples. = Koha plugin system = It seems that people are curious about our plugin system. It would be good to revive the discussion on new and more powerful plugin system. See the discussion on the koha-devel list at http://lists.koha-community.org/pipermail/koha-devel/2016-May/042673.html = Transactions & exceptions = It would be good to see some developers organise a brainstorming session on the different topics that Tomas raised on his email to koha-devel a few weeks ago. http://lists.koha-community.org/pipermail/koha-devel/2016- September/043032.html = Mana = Paul presented us Mana (https://lists.katipo.co.nz/pipermail/koha/2016-July/045739.html) at the beginning of July. Morgan finished her internship and will be there are the hackfest to show us what she developed. Testers and QAers would certainly be welcomed to test and review this feature. = Hea = Hea is a website (http://hea.koha-community.org) collecting usage statistics from different Koha installations around the world. This feature is not enabled by default but is very useful for developers and other people loving statistics (so librarians isn't it?). If it is not done yet, you should enable this feature to join the 496 libraries already registered! One enhancement is developed and waiting for testers: 14608 - HEA : add possibility of sharing usage statistics in Administration page and Web installer Hope to see you ready to fix, translate, test or QA patches (and drink beers when everything is done) this week! Cheers, Jonathan _______________________________________________ Koha-devel mailing list Koha-devel(a)lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/

2 2

Where do I add a topic for discussion in the next irc dev meeting?
by Barton Chittenden 23 Oct '16

23 Oct '16

Pardon my ignorance, but I'd like to add a topic for discussion in the next dev meeting, and I can't quite figure out where that should go -- the last dev meeting has come and gone: https://wiki.koha-community.org/wiki/Development_IRC_meeting_19_October_2016, but there's not another meeting scheduled to which I can add agenda items... also, is that the right way to do it, if the page did exist?

3 2

ID's in RESTful bodies
by Renvoize, Martin 19 Oct '16

19 Oct '16

A few of us(1) got together for a short 'Hangout' to talk about RESTful best practices today with the idea of debating out some best practices before presenting them to the wider group here and in the next irc dev meeting to add to the coding guidelines. In this short hangout we discussed "ID's in request bodies". It's a contentious issue in lots of RESTful circles, so we thought it worth getting into a coding guideline sooner rather than later to make sure we're consistent. TL:DR: Use the Swagger attribute 'readOnly' on id fields. This means id's are encouraged in responses but are banned in requests. There are lots of camps on this one, but the important thing is to be consistent across the api endpoints and ensure it's well documented. The swagger meaning of readOnly is well understood by rest users and thus accounts for the 'well documented' side, and implementing it using the Swagger plugin should mean minimal effort in ensuring it is consistent. 1) Jonathan D, Tomas C, Lari T, Kyle H and I *Martin Renvoize* Development Manager *T:* +44 (0) 1483 378728 *F:* +44 (0) 800 756 6384 *E:* martin.renvoize(a)ptfs-europe.com www.ptfs-europe.com <https://www.ptfs-europe.com> Registered in the United Kingdom No. 06416372 VAT Reg No. 925 7211 30 The information contained in this email message may be privileged, confidential and protected from disclosure. If you are not the intended recipient, any dissemination, distribution or copying is strictly prohibited. If you think that you have received this email message in error, please email the sender at info(a)ptfs-europe.com

1 0

What's on in Koha devel #6 - Special post hk16
by Jonathan Druart 19 Oct '16

19 Oct '16

Hello librarians and developers, On the same way as my previous email, I think it would be useful for everybody to get feedback on what was going on during the hackfest in Marseille last week. I plan to send the next "What's on in Koha devel" emails to both koha and koha-devel lists, if nobody disagrees (someone?). I think it would be helpful for developers not heavily involved in the development as well as techie librarians. Here are the different topics on which the developers have focused during this hackfest (as far I am aware of!): = Refactoring = Thanks to testers and QAers, a lot of things have moved in this area. Only 2 are waiting for QA and 9 are waiting to be pushed, and lot of others have been pushed! I will have to refill our bug tracker with new patches soon :) Unfortunately the move of the biblioitems.marcxml field to another table (bug 17196) did not interest anybody. It's however our first step toward new interesting features and don't necessarily depend on marc to store the records. = Elastic search = The mapping configuration page (bug 14899) have been pushed during this hackfest alongside some other bug fixes. A couple of UNIMARC users have submitted a patch for the authority mappings (bug 17373), now we need a couple of others to validate it! = Mana = Morgan presented us Mana and how it would help librarians to share data (subscriptions infos at the moment but could be extended to reports for instance), see bug 17373 if you are interested by testing these patches and/or understanding how it works. = RESTful API & ReactJS = The developer team works in the aim to submit a proposal for a full integration of our RESTful API, from the objects to the templates. We wanted to work all together on this problematic to show the community how easy and clean it can be. That means that we had a lot of passionated and intense discussions, as you can imagine, to reach a consensus on tiny things (variable's name) or big architecture problematics to simplify all the stack code. Basically we tried to get the simplest component as possible (the Koha::Cities modules and admin/cities.tt template) to make it an example to follow. For the RESTful API, all what we have is on bug 17428 (and dependencies). Yes, we know, it does not look like a lot for a whole week of work, but we needed to agree ;) The idea of having a small example gave us enough time to have a look at several things like exceptions, inconsistencies in our existing implementation, etc. About ReactJS, Kyle showed us how it could work on a small example (bug 17297) and then on a more complete example using the RESTful API endpoints (https://github.com/bywatersolutions/bywater-koha-devel/commits/cities_editor) If you are a developer or interested on this topic you should definitely have a look at these patches to know what could be the future of Koha ;) No decision has been taken so far and everything is here for discussion and example. During the next meeting, we will certainly talk about some minor but important details like "should PUT accept the id in the body". Stay tunes! Please complete this list if I forget something! The next dev meeting is on November 9th and the time has been set to 19UTC but is flexible. I'd like to try something new and get an answer to this survey: https://framadate.org/3dAEB8zqQLzzTptD from people wanting to attempt the dev meetings. The goal is not to get only 1 time slot but 2, in order to alternate between them. I repeat, you should only answer if you plan to attempt dev meetings :) Cheers, Jonathan

3 3

Reminder: General IRC meeting 19 October 2016, 10 UTC
by Jonathan Druart 18 Oct '16

18 Oct '16

Hi all, There is a general IRC meeting scheduled for tomorrow, 19 October, 10 UTC: https://wiki.koha-community.org/wiki/Development_IRC_meeting_19_October_2016 Check your timezone: https://www.timeanddate.com/worldclock/fixedtime.html?msg=Koha+Developers+I… Please make sure to add any topics you want to discuss to the agenda! Hope to see you there, Katrin^WJonathan

1 0

Git Server
by Brendan Gallagher 17 Oct '16

17 Oct '16

Hello All - Our hosting provider has alerted us to a problem on the host for git.koha-community.org. We are in the process of working with them to restore the server. We will let you all know when it has been restored. Thanks, Brendan -- --------------------------------------------------------------------------------------------------------------- Brendan A. Gallagher ByWater Solutions CEO

1 1

Status values in message_queue table
by Graham, Stephen 17 Oct '16

17 Oct '16

Hi All - doing: select status from message_queue group by status gives me 3 values: sent pending failed Are these 3 values the only value "allowed" in this field? Or, putting it another way, if messages are added to the table with a different status value would it cause problems in Koha? The reason I'm asking is: as soon a book with a reservation is returned on our selfcheck machines an email is added to the message_queue table with the status of pending. A lot can happen to this book before it actually gets on the reserve shelf, and may never actually get to the reserve shelf. If the book does go missing, gets hidden etc, the user with the reservation still gets an email telling them that the book is ready to collect (when it's not). To avoid this we are wondering if could stop this email from being sent (we only send out email notices once a day). We could delete the email from the message_queue table - I don't really like deleting stuff - or we could change the status. I guess I could change to "failed", but "cancelled" sounds better. I thinking of writing a bit of code to allow library staff to do this easily from the staff client - could I change the status to "cancelled" or is it better to stay with one of the original three? Cheers, Stephen Stephen Graham Library Technology Consultant Academic Resources Library and Computing Services University of Hertfordshire Hatfield AL10 9AB UK Tel. 01707 286111 Ext: 77751 Email s.graham4(a)herts.ac.uk

3 2

KohaCon17 status ?
by Paul Poulain 14 Oct '16

14 Oct '16

Hi all, Reading the minutes of the previous IRC discussions, I'm not sure that the KohaCon17 has been decided/validated yet. Am I wrong ? I'd like to point that, in Europe, there's a very important conference (ELAG, can be considered as the European Code4lib) that will take place in June 6th-9th (in Athens, Greece) having the KohaCon not set this week seems a good idea ;-) -- Paul Poulain, Associé-gérant / co-owner BibLibre, Services en logiciels libres pour les bibliothèques BibLibre, Open Source software and services for libraries

1 0