August 2016 - Koha-devel - lists.koha-community.org

biblioitems.marcxml & biblioitems.marc / HUGE performance issue !
by Paul Poulain 31 Aug '16

31 Aug '16

Hi all, Those days, we're working on a pretty large DB ( >1M biblio), for a customer that want to do many statistics on some fields. We discovered that something "simple" like: SELECT publicationyear, count(publicationyear) FROM biblioitems GROUP BY publicationyear; was giving *no result in 10mn*. This is a test DB, not optimized, but we were surprised by the results. After investigating we had the idea to create a biblioitems2 table with the same structure *EXCEPT MARCXML and MARC fields *launch the same SQL query : *result in 3seconds* !* *This could be reproduced on any query (on fields without index). I think it's because the innoDB is storing each line in one "object", so, even if you need only one column, you have to read everything. In our case, that was 12GB+ of data to read. biblioitems2 is just a few dozen MB. (all caching values are minimum and there's no index, so not involved in the results) MY CONCLUSIONS: * the biblioitems.marc field must be removed quickly: it's useless since years, and is only resulting in slowing things * the bilbioitems.marcxml field should be moved outside from this table. Something like biblio_blob, with biblionumber, biblioitemnumber and marcxml. When we need it, just join the tables. I'm almost sure it would have an important impact on Koha, as biblioitems table is called and used "everywhere". any opinion ? ** -- Paul Poulain, Associé-gérant / co-owner BibLibre, Services en logiciels libres pour les bibliothèques BibLibre, Open Source software and services for libraries

6 7

GoogleIndicTransliteration must die?
by Indranil Das Gupta 31 Aug '16

31 Aug '16

Hi all, Koha uses the deprecated (in 2011) Google Indic Transliterate v1 API to power the GoogleIndicTransliterate syspref. It seems internally the API calls to Googe Inputtools service over http! So again we are back in MIXED CONTENT mess and nothing much we can do about it, except move out of this API. transliteration.I.js:formatted:11055 Mixed Content: The page at 'https://rksmvvlibrary.in/cgi-bin/koha/opac-search.pl?q=%E0%A6%B6%E0%A7%8D%E…' was loaded over HTTPS, but requested an insecure script 'http://www.google.com/inputtools/request?text=shyama&ime=transliteration_en…'. This request has been blocked; the content must be served over HTTPS.Q.dh @ transliteration.I.js:formatted:11055Ww @ transliteration.I.js:formatted:4578Q.Ad @ transliteration.I.js:formatted:10982(anonymous function) @ transliteration.I.js:formatted:11237(anonymous function) @ transliteration.I.js:formatted:6670(anonymous function) @ transliteration.I.js:formatted:6059nq @ transliteration.I.js:formatted:2266(anonymous function) @ transliteration.I.js:formatted:6232a @ transliteration.I.js:formatted:6074 transliteration.I.js:formatted:10923 Uncaught TypeError: Cannot read property '0' of nullQ.Bh @ transliteration.I.js:formatted:10923Q.Mi @ transliteration.I.js:formatted:10862Mt.(anonymous function).j @ transliteration.I.js:formatted:8324(anonymous function) @ transliteration.I.js:formatted:3591 I saw this http://stackoverflow.com/questions/35238710/deprecated-google-api-will-not-… and this https://groups.google.com/forum/?fromgroups#!topic/google-ajax-search-api/9… Looks like its time to bid goodbye to GoogleIndicTransliteration syspref :-( -- Indranil Das Gupta L2C2 Technologies Phone : +91-98300-20971 Blog : http://blog.l2c2.co.in IRC : indradg on irc://irc.freenode.net Twitter : indradg

2 1

Interface patterns (tabs)
by David Cook 30 Aug '16

30 Aug '16

Hi Owen, Do you have any documentation for preferred us of jQuery tabs? I'm looking at https://wiki.koha-community.org/wiki/Interface_patterns but nothing is jumping out at me. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St Ultimo, NSW 2007 Australia Office: 02 9212 0899 Direct: 02 8005 0595

2 2

schema.koha-community.org
by Mark Tompsett 26 Aug '16

26 Aug '16

Greetings, Currently, it lists whatever is run as of a particular date. Is there a way to version it? There are currently 181 tables, not 175. So, this is clearly the difference between master and a release. https://schema.koha-community.org/{version}/... Just inquiring and thinking out loud. GPML, Mark Tompsett

4 3

Sending Mail...
by Mark Tompsett 25 Aug '16

25 Aug '16

Greetings, Jonathan Druart wrote on the Koha list: > When you claim issues, the email is sent directly and does not pass by > the message_queue table. This sounds like an area for massive refactorization: the standardization of sending mail. Does it not? GPML, Mark Tompsett

2 1

Git server downtime for maintenance, Aug 24, 2016
by Larry Baerveldt 24 Aug '16

24 Aug '16

Greetings, The Koha Community git server (git.koha-community.org) will be down for maintenance on Wed, Aug. 24, 2016, starting on or shortly after 7 pm PDT (2 am UTC). We cannot say for certain exactly how long the site will be down, but we hope it will be less than an hour. It would probably be best to avoid committing any updates to git during that hour. - Larry -- *Larry Baerveldt* Systems Administrator ByWater Solutions Support and Consulting for Open Source Software Headquarters: Santa Barbara, CA Office: Seattle, WA Phone/Fax (888)900-8944 http://bywatersolutions.com larry(a)bywatersolutions.com What is Koha? <http://bywatersolutions.com/what-is-koha/>

1 0

Reminder: Development IRC Meeting - 24 August 2016, 7 UTC
by Katrin Fischer 23 Aug '16

23 Aug '16

Hi all, just a quick reminder about tomorrow's Development IRC Meeting. The agenda and a link to the time converter for finding out your local meeting time can as always be found on the wiki: https://wiki.koha-community.org/wiki/Development_IRC_meeting_24_August_2016 Please add any topics you want to discuss! Hope to see you there, Katrin

1 0

Move the AV code to the Koha namespace
by Jonathan Druart 23 Aug '16

23 Aug '16

Hi devs, I have already started to move the authorised values related code to the Koha namespace, but to continue I'd like to get feedback on the method to continue. I have asked you in the last "what's on in koha-devel" email, but did not get any answer. Please see bug 15799 for the step-by-step plan. Cheers, Jonathan https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15799

2 1

Re: [Koha-devel] 16.05, zebra and jessie
by Paul A 23 Aug '16

23 Aug '16

At 03:57 PM 8/23/2016 +0100, Jonathan Druart wrote: >2016-08-23 15:43 GMT+01:00 Paul A <paul.a(a)navalmarinearchive.com>: > > At 09:21 AM 8/23/2016 -0400, Barton Chittenden wrote: > > > > Zebra tends to be I/O bound -- we've seen it write enormous .zrs files to > > disk (~16G/query on large libraries). BugÃ 13665 mentions that searches > > could be taking upwards of 40 seconds to complete -- I think that we've > seen > > searches time out and return no results at about 1 minute. > > > > > > When I looked into "speed" some time ago > > <http://navalmarinearchive.com/z_koha/>, it was the fact that multi-core > > processing was not possible (Intel 8-core I7 machine.) Speed is the only > > reason that we decided to stay with 3.8.24 rather than upgrade. > >Could you stop propagating this argument and have a look at what has >been done recently on the performance side? >You should reconsider your choice and have a look at 16.05, or at >least maintained releases. >Unless you backport the security / bug fixes by yourself? > >Did you compare with 16.05.x with memcached + plack? That's what you >should do, instead of comparing 3.8 vs 3.18 Yes, I did. Please see my email to this list Tue, 28 Jun 2016 20:46:04 -0400. (8 core Intel I7, 16 Gigs RAM, SSD) -- and it's still between three and four times slower. Slight difference, it was on Ubuntu 16.04, but with MySQL re-installed to the "older" standard. I only "propagated" the 3.18 comparison because (a) I wrote it up, fully annotated, and it has all the NYTProf analyses, and (b) it makes the point about (not) multi-threading. I did not do such a full analysis with 16.05. And I do backport (or otherwise) fix security issues if and when I become aware of them... Best -- Paul > > Best -- Paul > > > > > > Is it possible to tune Zebra's space/time optimizations in any way so that > > it doesn't write such large files to disk? > > > > On Tue, Aug 23, 2016 at 5:38 AM, Jonathan Druart > > <jonathan.druart(a)bugs.koha-community.org> wrote: > > See bug 13665 - Retrieve facets from zebra is slow > > To understand why and when use_zebra_facet=1 is slow > > > > 2016-08-22 21:31 GMT+01:00 Barton Chittenden <barton(a)bywatersolutions.com>: > >> I haven't run into the issue with the dashes in idzebra-2.0 2.0.59, but I > >> have run into this, when using ICU-Chains: > >> > >> Bug 16581 : ICU tokenization bug in idzebra-2.0 2.0.59-1 > >> URLÃ Ã Ã Ã : > >> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16581 > >> PriorityÃ : P5 - low > >> UrgencyÃ Ã : enhancement > >> StatusÃ Ã : NEW > >> > >> I also know that when use_zebra_facets was first introduced, it was *very* > >> slow -- I can't find any bugs about that though. It's possible that it got > >> so slow under idzebra-2.0 2.61 that the searches are timing out. > >> > >> It should be possible to set the logging for zebra so that you can see the > >> PQF queries: > >> > >> See > >> > >> Bug 15714 : Remove zebra.log from debian scripts and add optional log > >> levels > >> URLÃ Ã Ã Ã : > >> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15714 > >> PriorityÃ : P5 - low > >> UrgencyÃ Ã : enhancement > >> StatusÃ Ã : RESOLVED > >> > >> For setting the log levels > >> > >> And > >> http://koha.1045719.n5.nabble.com/Improving-Zebra-logging-td5861827.html > >> > >> For a general discussion of how to use them. > >> > >> ... This should give you some idea of what's failing, both in terms of the > >> dashes in 2.0.59 and the non-functional zebra facets under 2.0.61. > >> > >> My general feeling is that 2.0.59 is irredeemably broken by bug 16581, and > >> we need at least 2.0.60, but I don't have any experience with zebra > >> facets. > >> > >> --Barton > >> > >> > >> > >> On Mon, Aug 22, 2016 at 2:49 PM, Mark Tompsett <mtompset(a)hotmail.com> > >> wrote: > >>> > >>> Greetings, > >>> > >>> Similar problem. I hope someone has a better solution than setting it to > >>> 0. > >>> > >>> GPML, > >>> Mark Tompsett > >>> > >>> -----Original Message----- > >>> From: Philippe Blouin > >>> Sent: Monday, August 22, 2016 2:40 PM > >>> To: koha-devel(a)lists.koha-community.org > >>> Subject: [Koha-devel] 16.05, zebra and jessie > >>> > >>> Hello! > >>> > >>> We're trying to find the correction combination.Ã We're new on Jessie, > >>> so we still have some tweaking needed... > >>> > >>> - By default, we get zebra 2.00.59 installed on Jessie through the > >>> packages. > >>> - On 16.05, we get some very bad results in the search when the itemtype > >>> contains an hyphen (-), like 'A-DOC'. > >>> - So we installed zebra 2.00.62.Ã This fixes the search... > >>> - But now we do not have facets. > >>> - So we set <use_zebra_facets>0</use_zebra_facets> > >>> - And now we have facets.Ã But this feels... wrong? > > > >>> > >>> My dummy question: what is the supposedly correct version of Zebra on > >>> Jessie ? > >>> And we're we correct in setting the config to 0 ? > >>> > >>> Thanks > >>> Blou > >>> _______________________________________________ > >>> Koha-devel mailing list > >>> Koha-devel(a)lists.koha-community.org > >>> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > >>> website : http://www.koha-community.org/ > >>> git : http://git.koha-community.org/ > >>> bugs : http://bugs.koha-community.org/ > >>> > >>> _______________________________________________ > >>> Koha-devel mailing list > >>> Koha-devel(a)lists.koha-community.org > >>> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > >>> website : http://www.koha-community.org/ > >>> git : http://git.koha-community.org/ > >>> bugs : http://bugs.koha-community.org/ > >> > >> > >> > >> _______________________________________________ > >> Koha-devel mailing list > >> Koha-devel(a)lists.koha-community.org > >> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > >> website : http://www.koha-community.org/ > >> git : http://git.koha-community.org/ > >> bugs : http://bugs.koha-community.org/ > > > > > > _______________________________________________ > > Koha-devel mailing list > > Koha-devel(a)lists.koha-community.org > > http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > > website : http://www.koha-community.org/ > > git : http://git.koha-community.org/ > > bugs : http://bugs.koha-community.org/ > > > > --- > > Maritime heritage and history, preservation and conservation, > > research and education through the written word and the arts. > > <http://NavalMarineArchive.com> and <http://UltraMarine.ca> > > > > > > _______________________________________________ > > Koha-devel mailing list > > Koha-devel(a)lists.koha-community.org > > http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > > website : http://www.koha-community.org/ > > git : http://git.koha-community.org/ > > bugs : http://bugs.koha-community.org/ >_______________________________________________ >Koha-devel mailing list >Koha-devel(a)lists.koha-community.org >http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel >website : http://www.koha-community.org/ >git : http://git.koha-community.org/ >bugs : http://bugs.koha-community.org/ --- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>

1 0

Re: [Koha-devel] 16.05, zebra and jessie
by Paul A 23 Aug '16

23 Aug '16

At 09:21 AM 8/23/2016 -0400, Barton Chittenden wrote: >Zebra tends to be I/O bound -- we've seen it write enormous .zrs files to >disk (~16G/query on large libraries). BugÂ 13665 mentions that searches >could be taking upwards of 40 seconds to complete -- I think that we've >seen searches time out and return no results at about 1 minute. When I looked into "speed" some time ago <http://navalmarinearchive.com/z_koha/>, it was the fact that multi-core processing was not possible (Intel 8-core I7 machine.) Speed is the only reason that we decided to stay with 3.8.24 rather than upgrade. Best -- Paul >Is it possible to tune Zebra's space/time optimizations in any way so that >it doesn't write such large files to disk? > >On Tue, Aug 23, 2016 at 5:38 AM, Jonathan Druart ><<mailto:jonathan.druart@bugs.koha-community.org>jonathan.druart(a)bugs.koha-community.org> >wrote: >See bug 13665 - Retrieve facets from zebra is slow >To understand why and when use_zebra_facet=1 is slow > >2016-08-22 21:31 GMT+01:00 Barton Chittenden ><<mailto:barton@bywatersolutions.com>barton(a)bywatersolutions.com>: > > I haven't run into the issue with the dashes in idzebra-2.0 2.0.59, but I > > have run into this, when using ICU-Chains: > > > > Bug 16581 : ICU tokenization bug in idzebra-2.0 2.0.59-1 > > URLÂ Â Â Â : > <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16581>https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16581 > > PriorityÂ : P5 - low > > UrgencyÂ Â : enhancement > > StatusÂ Â : NEW > > > > I also know that when use_zebra_facets was first introduced, it was *very* > > slow -- I can't find any bugs about that though. It's possible that it got > > so slow under idzebra-2.0 2.61 that the searches are timing out. > > > > It should be possible to set the logging for zebra so that you can see the > > PQF queries: > > > > See > > > > Bug 15714 : Remove zebra.log from debian scripts and add optional log > levels > > URLÂ Â Â Â : > <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15714>https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15714 > > PriorityÂ : P5 - low > > UrgencyÂ Â : enhancement > > StatusÂ Â : RESOLVED > > > > For setting the log levels > > > > And > <http://koha.1045719.n5.nabble.com/Improving-Zebra-logging-td5861827.html>http://koha.1045719.n5.nabble.com/Improving-Zebra-logging-td5861827.html > > > > For a general discussion of how to use them. > > > > ... This should give you some idea of what's failing, both in terms of the > > dashes in 2.0.59 and the non-functional zebra facets under 2.0.61. > > > > My general feeling is that 2.0.59 is irredeemably broken by bug 16581, and > > we need at least 2.0.60, but I don't have any experience with zebra facets. > > > > --Barton > > > > > > > > On Mon, Aug 22, 2016 at 2:49 PM, Mark Tompsett > <<mailto:mtompset@hotmail.com>mtompset(a)hotmail.com> wrote: > >> > >> Greetings, > >> > >> Similar problem. I hope someone has a better solution than setting it to > >> 0. > >> > >> GPML, > >> Mark Tompsett > >> > >> -----Original Message----- > >> From: Philippe Blouin > >> Sent: Monday, August 22, 2016 2:40 PM > >> To: > <mailto:koha-devel@lists.koha-community.org>koha-devel(a)lists.koha-community.org > >> Subject: [Koha-devel] 16.05, zebra and jessie > >> > >> Hello! > >> > >> We're trying to find the correction combination.Â We're new on Jessie, > >> so we still have some tweaking needed... > >> > >> - By default, we get zebra 2.00.59 installed on Jessie through the > >> packages. > >> - On 16.05, we get some very bad results in the search when the itemtype > >> contains an hyphen (-), like 'A-DOC'. > >> - So we installed zebra 2.00.62.Â This fixes the search... > >> - But now we do not have facets. > >> - So we set <use_zebra_facets>0</use_zebra_facets> > >> - And now we have facets.Â But this feels... wrong? > >> > >> My dummy question: what is the supposedly correct version of Zebra on > >> Jessie ? > >> And we're we correct in setting the config to 0 ? > >> > >> Thanks > >> Blou > >> _______________________________________________ > >> Koha-devel mailing list > >> > <mailto:Koha-devel@lists.koha-community.org>Koha-devel(a)lists.koha-community.org > >> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > >> website : <http://www.koha-community.org/>http://www.koha-community.org/ > >> git : <http://git.koha-community.org/>http://git.koha-community.org/ > >> bugs : <http://bugs.koha-community.org/>http://bugs.koha-community.org/ > >> > >> _______________________________________________ > >> Koha-devel mailing list > >> > <mailto:Koha-devel@lists.koha-community.org>Koha-devel(a)lists.koha-community.org > >> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > >> website : <http://www.koha-community.org/>http://www.koha-community.org/ > >> git : <http://git.koha-community.org/>http://git.koha-community.org/ > >> bugs : <http://bugs.koha-community.org/>http://bugs.koha-community.org/ > > > > > > > > _______________________________________________ > > Koha-devel mailing list > > > <mailto:Koha-devel@lists.koha-community.org>Koha-devel(a)lists.koha-community.org > > http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > > website : <http://www.koha-community.org/>http://www.koha-community.org/ > > git : <http://git.koha-community.org/>http://git.koha-community.org/ > > bugs : <http://bugs.koha-community.org/>http://bugs.koha-community.org/ > > >_______________________________________________ >Koha-devel mailing list >Koha-devel(a)lists.koha-community.org >http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel >website : http://www.koha-community.org/ >git : http://git.koha-community.org/ >bugs : http://bugs.koha-community.org/ --- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>

2 1