March 2017 - Koha-devel - lists.koha-community.org

Expected behaviod when printing serials - add reserve?
by Jonathan Druart 25 May '17

25 May '17

Hi devs, I found an obvious bug in serials/routing-preview.pl. If the script is called with ok=1 (hum... yes...), the script will add or modify a hold placed on the bibliographic records existing in the routing list. So from /serials/serials-collection.pl?subscriptionid=1, click on "Print list" you get a pop-up with a print and close button. When the pop-up is generated, we already have the ok=1 and the go through: 66: my ($count2,@bibitems) = GetBiblioItemByBiblioNumber($biblio); Then later: 96 AddReserve($branch,$routing->{borrowernumber},$biblio,\@bibitems,$routing->{ranking}, undef, undef, $notes,$title); The problem is that GetBiblioItemByBiblioNumber will always return only 1 value (for obvious reasons) and @bibitems will always be empty => reserve is placed at bib level The thing is that this bug exists for ages (at least 2006, maybe always). What is the expected behaviour? No need to tell you that I was going to kick GetBiblioItemByBiblioNumber when I found that bug. Cheers, Jonathan

3 2

Single character searches are slow when queryfuzzy and/or querystemming are enabled.
by Barton Chittenden 09 May '17

09 May '17

I've got an issue that we've seen periodically at ByWater; I want to file a bug, but I don't have a clear idea of how to replicate the issue, so I'm trying to solicit information from the community on how to narrow the scope of the problem and/or measure the performance problems that I'm seeing. The problem is that keyword searches (as you'd see from an un-modified masthead search on the opac, or the "search the catalog" search on the staff client) are markedly slow on some sites when a) The search term contains single letter words like "a" in "Once Upon A Time" and b) One or both of the QueryFuzzy or QueryStemming system preferences is enabled. We first ran across this issue when we moved a number our Koha instances to slower drives, and found that some (but not all) would time out when searching for "Once upon a time", "A swiftly tilting planet", "Hitchiker's Guide to the Galaxy" (where the 's' after the apostrophe is counted as a single word). I turned on 'request' logging in zebra, here's the query that timed out: find @attrset Bib-1 @or @or @or @or @or @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 "once upon a time" @attr 1=4 @attr 4=1 @attr 6=3 @attr 9=28 @attr 2=102 "once upon a time" @attr 1=36 @attr 4=1 @attr 9=26 @attr 2=102 "once upon a time" @attr 1=4 @attr 4=6 @attr 9=24 @attr 2=102 "once upon a time" @attr 4=6 @attr 5=103 @attr 9=16 @attr 2=102 "once upon a time" @attr 4=6 @attr 5=1 @attr 9=14 @attr 2=102 "onc? upon? a time? " @attr 4=6 @attr 9=14 @attr 2=102 "once upon a time" Sent searchRequest. I ran the PQF in zebra: Z> base biblios Z> find @attrset Bib-1 @or @or @or @or @or @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 "once upon a time" @attr 1=4 @attr 4=1 @attr 6=3 @attr 9=28 @attr 2=102 "once upon a time" @attr 1=36 @attr 4=1 @attr 9=26 @attr 2=102 "once upon a time" @attr 1=4 @attr 4=6 @attr 9=24 @attr 2=102 "once upon a time" @attr 4=6 @attr 5=103 @attr 9=16 @attr 2=102 "once upon a time" @attr 4=6 @attr 5=1 @attr 9=14 @attr 2=102 "onc? upon? a time? " @attr 4=6 @attr 9=14 @attr 2=102 "once upon a time" Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 119, setno 1 SearchResult-1: term=once upon a time cnt=3, term=once upon a time cnt=5, term=once cnt=94, term=upon cnt=63, term=a cnt=4876, term=time cnt=515, term=once cnt=168, term=upon cnt=125, term=a cnt=19801, term=time cnt=1507, term=once cnt=9830, term=upon cnt=960, term=a cnt=82237, term=time cnt=8524, term=onc cnt=1915, term=upon cnt=914, term=a cnt=82237, term=time cnt=7709, term=once cnt=1302, term=upon cnt=914, term=a cnt=68283, term=time cnt=5442 records returned: 0 Elapsed: 98.163583 The elapsed time is just under 100 seconds; I think Koha times out after 60. We looked at the disk I/O and found that the disk where the searches were occurring was getting hammered, so we migrated the sites that were having issues onto a server that had a much faster drive, and this did, at least solve the time-out issue. The issue keeps on popping up, however -- if not in actual timeouts, then at least in complaints of slowness in search. My hypothesis is that either QueryFuzzy or QueryStemming is expanding the one letter words, i.e. a search for "a" is returning results for either all words that start with "a" or all words contaning "a", and that all of these results are written to disk before any further filtering is done. When we were seeing timeout issues, I'm not clear on which sites were likely to have issues -- sites with a very small number of bibs didn't have issues, but the problem wasn't solely by collection size. So here's what I'd like to know, for the purposes of further testing, and/or to be able to replicate the issue: 1) Is there anyone who knows all about PQF who can tell me why the query above would run so slowly? 2) Is there part of the PQF that seems to be behaving particularly badly? Which parts of the query are returning "term=a cnt=82237", and can we avoid having that called twice? 3) How do I go about constructing a data set that illustrates this problem? 4) What's the best way to measure the performance problems and/or define what the problem is when I file a bug? 5) Are there any work-arounds, given that certain sites really want query stemming and queryfuzzy enabled? Thanks, --Barton

2 1

Task schedulers and message queues for Koha
by David Cook 04 May '17

04 May '17

Hi all, In 2016, I worked on a Koha task scheduler for downloading and importing records via OAI-PMH. I have code which works, but it's lacking test coverage and I'm unsure that it will make it through QA and be accepted. I recall Chris Cormack suggesting I look at Gearman (http://gearman.org/) instead, and it looks pretty good at a glance, although Andreas and I had talked about having more control over the workers than Gearman seems to offer. Plus, it added another dependency to Koha where people already struggle with dependencies. Martin suggested that there were a lot of other implementations out there, but I haven't really found much else that provides everything we want out of the box. The one I have found is Celery (http://www.celeryproject.org/) which looks like exactly what I want I think, but it requires Python for its server and workers, and it requires a message broker like RabbitMQ or Redis. Lots of extra dependencies. Since my goal is to get our code into Koha, I really want to know what will work for people. Are people happy with a home grown solution? It's not really that complicated. My current version is essentially a task scheduler which forks a worker on demand when it's time to run a task (up to a configurable max of X tasks so you don't kill your server). It lets you submit tasks, tell the scheduler to start/schedule them, and you can even tell in progress tasks to stop (by having the scheduler tell the worker to stop and the worker decides where in its task it checks for stop commands from the scheduler). As I try to add test coverage and make this scheduler more palatable, I find myself thinking about the code more like Koha::Scheduler and Koha::Queue, and using the more scalable worker model used by others. The scheduler daemon would listen on a socket for tasks, it would create a Koha::Scheduler instance which would enqueue tasks to run once that task's time was met or exceeded. Now depending on the architecture. you could have a separate daemon or the same daemon with a Koha::Queue instance. It would accept tasks/messages from the scheduler, and it would dequeue tasks to available workers - which are separate processes - have previously registered against particular queues. In this way, you can have a oaipmh-download queue, oaipmh-import queue, a email-report queue, etc. I suspect that we could make use of Koha::Scheduler and Koha::Queue throughout much of Koha for doing background tasks. If we don't want to reinvent the wheel with Koha::Queue, we could use something like RabbitMQ. But I think we need *something*. I'm open to ideas. I already have the OAI-PMH download and OAI-PMH import handled. That's the easy part. Any worker can do that. The hard part is figuring out how the Koha Community will take up a task scheduler. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595

4 8

Koha + Elasticsearch 5.x dependency packages
by Mirko Tietgen 28 Apr '17

28 Apr '17

Hi, it looks like I am done submitting and backporting dependencies for Koha and Elasticsearch 5.x. I ran it in a test repo and it works as far as I can see. Adding them to the unstable repository may or may not break Kohadevbox and other setups, so I thought I'd give you a warning before I do that. I will add the following packages: libapp-cmd-perl_0.331-1~kohadev1 libbusiness-isbn-perl_3.003-1~kohadev1 libcatmandu-marc-perl_1.09-1~kohadev1 libcatmandu-perl_1.0304-2~kohadev1 libcatmandu-store-elasticsearch-perl_0.0507-1~kohadev1 libcpanel-json-xs-perl_3.0225-1~kohadev1 liblog-any-perl_1.045-1~kohadev1 libmarc-lint-perl_1.50-1~kohadev1 libmoox-aliases-perl_0.001006-1~kohadev1 libmoox-role-logger-perl_0.005-1~kohadev1 libnamespace-clean-perl_0.27-1~kohadev1 libsearch-elasticsearch-perl_5.01-1~kohadev1 libmarc-spec-perl_0.1.2-1~kohadev1 libparser-mgc-perl_0.15-1~kohadev1 libref-util-perl_0.113-1~kohadev1 libtext-hogan-perl_1.04-1~kohadev1 liburi-template-perl_0.22-0.1~kohadev1 Tomas (and other people with Kohadevbox knowledge), I suppose you need to make changes to not CPAN stuff anymore? To pull in the new packages, it should be enought to apt-get install libcatmandu-store-elasticsearch-perl libcatmandu-marc-perl once they are in the repository. I plan to have the packages in the unstable repository next monday. Cheers, Mirko -- Mirko Tietgen mirko(a)abunchofthings.net http://koha.abunchofthings.net http://meinkoha.de

3 4

Documenting notices and scripts
by Barton Chittenden 01 Apr '17

01 Apr '17

I've been working on documenting all of the possible notices in the letter table, in terms of letter_code, available markup, and which tables we can pull data from. this is collected here: https://wiki.koha-community.org/wiki/Customising_Notices_and_Slips#Notice_S… I updated the wiki page this morning, everything is up-to-date with master... but there's one thing I don't understand: SendAlerts. It looks like the letter code is passed in via cgi param, and I don't know what letter_codes are available for the calls in C4/Serials.pm, acqui/ lateorders.pl and serials/claims.pl Searching for enlightenment, --Barton

1 0

Koha Community Newsletter: March 2017
by kohanews 01 Apr '17

01 Apr '17

Fellow Koha users: The Koha Community Newsletter for March 2017 is here: https://koha-community.org/koha-community-newsletter-march-2017/ Many thanks to the folks who submitted articles and news to this month's newsletter. Please feel free to email me with any corrections or suggestions. Thanks! -- Chad Roseburg Editor, Koha Community Newsletter

1 0

[Koha] Koha 3.22.19 security release
by Julian Maurice 31 Mar '17

31 Mar '17

The Koha community is proud to announce the release of Koha 3.22.19. This is a security release and contains several bugfixes. If you are running 3.22.x it is strongly recommended to upgrade. The full release notes are available at https://koha-community.org/koha-3-22-19-security-release/

1 0

Koha 16.11.06 released
by Katrin 31 Mar '17

31 Mar '17

The Koha community is proud to announce the release of Koha 16.11.06. This is a maintenance release of 16.11 and contains 9 bugfixes. The full release notes are available at https://koha-community.org/koha-16-11-06-release/ Packages will be available soon.

2 1

MySQL silently truncating strings due to 3 byte limit in 'utf8' charset
by David Cook 31 Mar '17

31 Mar '17

Hi all, I ran into a problem this afternoon where MySQL was silently truncating my UTF8 encoded string in biblio_metadata.metadata and biblio.notes. Admittedly, the string had some funny looking data*, but it was all valid UTF8. I had zero problems with XML::LibXML or MARC::Record. However, whenever I would use DBI or the mysql command line client, I’d get a truncated string**. This meant a completely invalid XML record which is irretrievable from the database using Koha’s XML handling methods. However, I eventually found https://bugs.mysql.com/bug.php?id=53005 where they state that the MySQL ‘utf8’ charset has a 3 byte limit (https://dev.mysql.com/doc/refman/5.7/en/charset-unicode-utf8.html) That tends to be fine in most cases, but 𝔤 is 4 bytes (f0 9d 94 a4 http://dev.networkerror.org/utf8/?start=120100 <http://dev.networkerror.org/utf8/?start=120100&end=120200&cols=4&show_uni_i…> &end=120200&cols=4&show_uni_int=on&show_uni_hex=on&show_html_ent=on&show_raw_hex=on&show_raw_bin=on). It was on this character that the truncation happened. Apparently, there is a 4 byte UTF-8 charset called utf8mb4: https://dev.mysql.com/doc/refman/5.7/en/charset-unicode-utf8mb4.html. If you “SET NAMES = ‘utf8mb4’” for your client and set the charset for your database column, that will allow you to correctly insert the data. So the truncation doesn’t generate an error but it does generate a warning, which is much more obvious in the mysql CLI client than when using DBI: MariaDB [devkohadcook]> update biblio set abstract = 'We introduce a new perspective and a generalization of spectral networks for 4d N=2" style="position: relative;" tabindex="0" id="MathJax-Element-1-Frame" class="MathJax">=2 theories of class S" style="position: relative;" tabindex="0" id="MathJax-Element-2-Frame" class="MathJax"> associated to Lie algebras g=An" style="position: relative;" tabindex="0" id="MathJax-Element-3-Frame" class="MathJax">𝔤=An, Dn" style="position: relative;" tabindex="0" id="MathJax-Element-4-Frame" class="MathJax">Dn, E6" style="position: relative;" tabindex="0" id="MathJax-Element-5-Frame" class="MathJax">E6, and E7" style="position: relative;" tabindex="0" id="MathJax-Element-6-Frame" class="MathJax">E7. Spectral networks directly compute the BPS spectra of 2d theories on surface defects coupled to the 4d theories. A Lie algebraic interpretation of these spectra emerges naturally from our construction, leading to a new description of 2d-4d wall-crossing phenomena. Our construction also provides an efficient framework for the study of BPS spectra of the 4d theories. In addition, we consider novel types of surface defects associated with minuscule representations of g" style="position: relative;" tabindex="0" id="MathJax-Element-7-Frame" class="MathJax">𝔤.' where biblionumber = 50314; Query OK, 0 rows affected, 1 warning (0.00 sec) Rows matched: 1 Changed: 0 Warnings: 1 Because it doesn’t raise an exception, I’d say it’s silent. You have to check for it. Using mysql, you can send “SHOW WARNINGS” after the insert query and you can see something like: +---------+------+--------------------------------------------------------------------------------+ | Level | Code | Message | +---------+------+--------------------------------------------------------------------------------+ | Warning | 1366 | Incorrect string value: '\xF0\x9D\x94\xA4=A...' for column 'abstract' at row 1 | +---------+------+--------------------------------------------------------------------------------+ 1 row in set (0.01 sec) With DBI, it’s harder: http://stackoverflow.com/questions/7946655/dbi-perl-logging-mysql-warnings. Apparently you can use $dbh->{mysql_warning_count} to check for warnings, and then send “SHOW WARNINGS” via the same database handle and that should give you your warning result. Or you can run MySQL in “traditional” mode which treats warnings as errors. So this seems like an edgecase… it’s some wonky data (albeit valid UTF8 data) in an otherwise English record. However, according to MySQL (I haven’t verified for myself), Chinese, Japanese, and Korean sometimes use 4 bytes for characters. So people using these languages with Koha could be getting truncated data without realising that’s what is happening or why. I’ve tried out utf8mb4 and it works great. I think the downside is that it uses more space for certain types of columns because it needs to allocate more space for that 4th byte. I don’t know whether we should consider switching from utf8 to utf8mb4, so that we’re inclusive of CJK and comply with RFC3629 which states that there’s a 4 byte maximum. I think it’s worth discussion though. This wouldn’t just affect libraries in Asia. It would affect any library which stores records with CJK characters in a MARC 880 “Alternate Graphic Representation” field. I’m at the end of my Friday work day, so I’m going to leave it there, but food for thought! *Aforementioned strange data: We introduce a new perspective and a generalization of spectral networks for 4d N=2" style="position: relative;" tabindex="0" id="MathJax-Element-1-Frame" class="MathJax">=2 theories of class S" style="position: relative;" tabindex="0" id="MathJax-Element-2-Frame" class="MathJax"> associated to Lie algebras g=An" style="position: relative;" tabindex="0" id="MathJax-Element-3-Frame" class="MathJax">𝔤=An, Dn" style="position: relative;" tabindex="0" id="MathJax-Element-4-Frame" class="MathJax">Dn, E6" style="position: relative;" tabindex="0" id="MathJax-Element-5-Frame" class="MathJax">E6, and E7" style="position: relative;" tabindex="0" id="MathJax-Element-6-Frame" class="MathJax">E7. Spectral networks directly compute the BPS spectra of 2d theories on surface defects coupled to the 4d theories. A Lie algebraic interpretation of these spectra emerges naturally from our construction, leading to a new description of 2d-4d wall-crossing phenomena. Our construction also provides an efficient framework for the study of BPS spectra of the 4d theories. In addition, we consider novel types of surface defects associated with minuscule representations of g" style="position: relative;" tabindex="0" id="MathJax-Element-7-Frame" class="MathJax">𝔤. ** Truncated string: We introduce a new perspective and a generalization of spectral networks for 4d N=2" style="position: relative;" tabindex="0" id="MathJax-Element-1-Frame" class="MathJax">=2 theories of class S" style="position: relative;" tabindex="0" id="MathJax-Element-2-Frame" class="MathJax"> associated to Lie algebras g=An" style="position: relative;" tabindex="0" id="MathJax-Element-3-Frame" class="MathJax">𝔤 David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595

3 4

Development/ QA protocol question
by Cab Vinton 30 Mar '17

30 Mar '17

Hi, All -- A small group of small NH libraries has co-sponsored a small enhancement to get its feet wet in the Koha development world: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14365 As I understand it, once a patch has been signed off & passed on to QA, it can hit a bottleneck as there are limited number of QA Team Members. To make this as positive experience as possible for our informal consortium & to increase the chances of similar collaborations in the future, it would help if we can move through the development process as quickly as possible. So here's my question -- For sponsored (i.e., funded) projects, would it make sense to include funds to pay for the cost of a QA Team Member's time to review a patch? I'm thinking the financial commitment would be one way of boosting a patch's status in the queue. Hounding the developer is another option, of course, but that's unpleasant for both sides :-) I realize the QA person often, & perhaps preferably, will work for a different firm. And there may be other complicating factors I'm not aware of. Grateful for any thoughts from folks on the programming side of things. Cheers, Cab Vinton, Director Plaistow Public Library Plaistow, NH

5 16