- Koha-devel - lists.koha-community.org

Re: [Koha-devel] Search speed
by Paul A 05 Feb '15

05 Feb '15

At 10:03 AM 2/5/2015 +0100, Jonathan Druart wrote: >I have filled bug 13665 yesterday >http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13665 Thanks. I had come to that same conclusion concerning resultset_record (see my email 9 Jan to this list.) For anyone wanting to start chasing this down, I noted: ZOOM modules: We have Net::Z3950::ZOOM (v1.26) Ubuntu libnet-z3950-zoom-perl (latest.) Includes libidzebra-2.0-mod-alvis, but there are two different files named ZOOM.pm me@hood:/$ cat -n /usr/lib/perl5/ZOOM.pm 1110 lines me@hood:/$ cat -n /usr/lib/perl5/Net/Z3950/ZOOM.pm 194 lines Debug: "book mark" 'f ZOOM' both choose the short one: 'b 152' therefore force 'f /usr/lib/perl5/ZOOM.pm' 'b 739' 'f /usr/share/koha/lib/C4/Auth.pm' 'b 442' return to 'f opac-search.pl' But from then on, chasing individual lines/functions (s, n, r) became frustrating, and a dump (T) was unmanageable. Again, thanks -- Paul >2015-02-04 22:22 GMT+01:00 Paul A <paul.a(a)navalmarinearchive.com>: > > At 02:02 PM 2/2/2015 +0000, Tomas Cohen Arazi wrote: > >> > >> Paul, while you are running your tests, could you please compare this both > >> scenarios: > >> - <use_zebra_facets>1</use_zebra_facets> > >> - <use_zebra_facets>0</use_zebra_facets> > >> Thanks > >> PS: I have found that searchResults is too slow too. > > > > > > I have posted notes, results and some NYTProf examples at > > <http://navalmarinearchive.com/z_koha/> -- the results are repeatable (I > > must have run NYTProf nearly 200 times.) > > > > Provisional conclusions are that <use_zebra_facets> = 1 (true) is not > usable > > in production for a search that finds > 20k results. For smaller > results, it > > might be acceptable, but is still appreciably slower than 3.08. Under the > > best circumstances (0-5 results) 3.08 responds in < 0.7 secs, 3.18 in 1.3 > > secs. At the other extreme (25k results), 3.08 requires 1.07 secs, > 3.18 21+ > > secs. > > > > I have defined an "optimal" 3.18 combination (see green line, Graph 2) > which > > runs at about half the speed of 3.08, but does not give 'Series', 'Places' > > and 'Titles' in the facets. > > > > One point that jumped out is that a best case 3.08 search executes 431207 > > statements and 76556 subroutine calls in 297 source files and 81 string > > evals whereas a worst case 3.18 executes 1238167 statements and 309905 > > subroutine calls in 591 source files and 132 string evals. (Even the worst > > case 308 appears much more efficient than the best case 3.18.) > > > > The use of -T to start Zebra is non-conclusive -- it vaguely slows down > > smaller result sets and vaguely speeds up larger result sets (testing is > > repeatable.) Robin was of course quite correct in his recent email, but I > > cannot explain even the minor variations experienced. > > > > I'm not at all certain how to file this as a bug, so will await any > > comments. > > > > Best -- Paul > > > > > > _______________________________________________ > > Koha-devel mailing list > > Koha-devel(a)lists.koha-community.org > > http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > > website : http://www.koha-community.org/ > > git : http://git.koha-community.org/ > > bugs : http://bugs.koha-community.org/ --- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>

1 0

Re: [Koha-devel] Search speed
by Paul A 05 Feb '15

05 Feb '15

At 02:02 PM 2/2/2015 +0000, Tomas Cohen Arazi wrote: >Paul, while you are running your tests, could you please compare this both >scenarios: >- <use_zebra_facets>1</use_zebra_facets> >- <use_zebra_facets>0</use_zebra_facets> >Thanks >PS: I have found that searchResults is too slow too. I have posted notes, results and some NYTProf examples at <http://navalmarinearchive.com/z_koha/> -- the results are repeatable (I must have run NYTProf nearly 200 times.) Provisional conclusions are that <use_zebra_facets> = 1 (true) is not usable in production for a search that finds > 20k results. For smaller results, it might be acceptable, but is still appreciably slower than 3.08. Under the best circumstances (0-5 results) 3.08 responds in < 0.7 secs, 3.18 in 1.3 secs. At the other extreme (25k results), 3.08 requires 1.07 secs, 3.18 21+ secs. I have defined an "optimal" 3.18 combination (see green line, Graph 2) which runs at about half the speed of 3.08, but does not give 'Series', 'Places' and 'Titles' in the facets. One point that jumped out is that a best case 3.08 search executes 431207 statements and 76556 subroutine calls in 297 source files and 81 string evals whereas a worst case 3.18 executes 1238167 statements and 309905 subroutine calls in 591 source files and 132 string evals. (Even the worst case 308 appears much more efficient than the best case 3.18.) The use of -T to start Zebra is non-conclusive -- it vaguely slows down smaller result sets and vaguely speeds up larger result sets (testing is repeatable.) Robin was of course quite correct in his recent email, but I cannot explain even the minor variations experienced. I'm not at all certain how to file this as a bug, so will await any comments. Best -- Paul

3 2

Zebra hyphen truncating search
by David Cook 05 Feb '15

05 Feb '15

Hi Colin! I stumbled across an email you sent to the Indexdata Zebralist in 2013: http://lists.indexdata.dk/pipermail/zebralist/2013-August/002576.html Were you ever able to solve those problems? Also, what version of Zebra were you running? I've noticed this problem with Zebra using 2.0.59, but I haven't been able to produce it using Zebra 2.0.47. I had the exact same configuration files, MySQL database, and Zebra indexes. Here's an example: 2.0.59: Z> f UPA-CE14060101 Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 12325, setno 1 SearchResult-1: term=UPA cnt=12325 records returned: 0 Elapsed: 0.021128 2.0.47: Z> f UPA-CE14060101 Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 1, setno 3 SearchResult-1: term=UPA cnt=12325, term=CE14060101 cnt=1 records returned: 0 Elapsed: 0.015657 -- It looks like the packages in Debian 7 and Ubuntu 14.04 are stuck back at Zebra 2.0.44, which is from 2010. However, Indexdata apparently has newer deb packages for both Debian and Ubuntu: http://ftp.indexdata.dk/pub/zebra/. In fact, the server I'm having problems with appears to have installed Zebra 2.0.59 using the following "/etc/apt/sources.list.d/indexdata.list" pointing to "deb http://ftp.indexdata.dk/debian squeeze main". I'm guessing that you were using a Zebra version that was higher than 2.0.47? Possibly a deb package installed from Indexdata's repos? I know Tomas has had good luck contacting Indexdata folk. maybe he has some ideas about how to find a solution to this issue? David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007

3 4

Re: [Koha-devel] Zebra hyphen truncating search Part Deux
by David Cook 04 Feb '15

04 Feb '15

Robin just pointed out that Debian Jessie (8) will be switching to 2.0.59 and I noticed that Ubuntu Vivid (15.04) will as well. That means that Koha package installs that use ICU on these newer OSes will start running into this problem as well. (Actually, most newer OSes will have this issue. OpenSuse 13.2 released in November 2014 is using 2.0.59.) So... that's not good. We first ran into this problem because of a barcode search "ABC-12345" was returning all barcodes that started with "ABC". However, you'll run into this problem for a lot of keyword and title searches. For instance "Mont-Royal" will return all records containing "Mont". Note that this is only for ICU. If you're using CHR, "Mont-Royal" will just get you records that have "Mont" and "Royal" in them (since Zebra/YAZ breaks on hyphens when using CHR character mapping). Anyway, I've sent an email to Indexdata, so I'm going to try pursuing this with them as much as possible... David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007 > -----Original Message----- > From: Colin Campbell [mailto:colin.campbell@ptfs-europe.com] > Sent: Tuesday, 3 February 2015 8:03 PM > To: David Cook > Cc: 'Colin Campbell'; koha-devel(a)lists.koha-community.org > Subject: Re: Zebra hyphen truncating search > > On Tue, Feb 03, 2015 at 05:16:05PM +1100, David Cook wrote: > > Hi Colin! > > > > > > > > I stumbled across an email you sent to the Indexdata Zebralist in 2013: > > http://lists.indexdata.dk/pipermail/zebralist/2013-August/002576.html > > > > > > > > Were you ever able to solve those problems? > > I think its still the case. It can be further complicated by what the settings in > the system are and how Search.pm interprets them in constructing the > search. Especially the relevancy options. > > > > Also, what version of Zebra were you running? I've noticed this > > problem with Zebra using 2.0.59, but I haven't been able to produce it > > using Zebra 2.0.47. I had the exact same configuration files, MySQL > > database, and Zebra indexes. Here's an example: > It would be the current releases from indexdata on Debian and also built > from source on Fedora. The indexdata debian packages track the source > closely. > > Cheers > Colin > > -- > Colin Campbell > Chief Software Engineer, > PTFS Europe Limited > Content Management and Library Solutions > +44 (0) 800 756 6803 (phone) > +44 (0) 7759 633626 (mobile) > colin.campbell(a)ptfs-europe.com > skype: colin_campbell2 > > http://www.ptfs-europe.com

1 0

Dev meeting February 5
by Tomas Cohen Arazi 02 Feb '15

02 Feb '15

Hi, I'm calling for a dev meeting next thursday. We'll be back to the two-part meeting, so everyone can attend. The agenda will be here, but it is now just a copy of the last meeting's: http://wiki.koha-community.org/wiki/Development_IRC_meeting_5_February_2015 Thanks! Tomas

1 0

Search speed
by Paul A 02 Feb '15

02 Feb '15

I have progressed a little further on this subject using multiple parallel tests for from 0 to ~25k biblios on a new, relatively fast server. In brief: 3.08 averages 0.88 secs per search 3.18 averages 5.06 secs for the same searches, but... 3.18 averages only 1.52 secs if I turn off facets. Facets appear to have a huge overhead in various components of Zebra, to the point where a single core of a multiple processor CPU gets overwhelmed, but does not share the load with other totally idle cores. Intel tell me that this is not fixable through any hardware/BIOS config and they are not aware of a Linux "standard library" that deals with processor affinity. I've searched around a little and found mentions for c and FORTRAN (marvelous old memories for me!) but so far nothing totally specific for Perl. taskset and sched_setaffinity might be usable at system level, and I might have a look at pthread_setaffinity_np.3.gz I've posted test results, notes and three sets of NYTProf analysis at <http://navalmarinearchive.com/z_koha/> and would appreciate if anyone has any suggestions or comparative tests and outcomes. Thanks and best regards -- Paul

5 7

New features for Dublin Core metadata
by Hector Castro 30 Jan '15

30 Jan '15

Hi, Koha Developers. I wrote a Dublin Core metadata support to export in XML and RDF formats. This code follow the Dublin Core Specifications at: < http://dublincore.org/specifications/> specially DC-XML, Simple DC-RDF, and DC-RDF recommendations. Aditionally support OAI-DC schema at < http://www.openarchives.org/OAI/2.0/oai_dc.xsd>. RDF can be exported as multiple formats like: rdfxml, rdfjson, ntriples, turtle, etc. It was neccesary to create a custom module (C4::DublinCoreTransformer) to transform the object created by DublinCore::Record and MARC::Crosswalk::DublinCore. The module returns all the formats as string. I file a bug to test it at: < http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13642> Regards, -- Atte, Héctor Eduardo Castro Avalos Universidad de El Salvador Facultad Multidisciplinaria Oriental

1 0

Template Toolkit BLOCK in INCLUDE file
by David Cook 29 Jan '15

29 Jan '15

Hi all: I was looking at defining some TT BLOCKs in an include file which I would INCLUDE into my main template and then PROCESS as needed (as part of bug 13631 http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13631) However, I noticed that you can't access a BLOCK in an included file. I noticed some seemingly non-functional examples already in the Koha codebase. For instance, in "koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-auth-detail.tt" we try to process the blocks from the include "/koha-tmpl/opac-tmpl/bootstrap/en/includes/opac-authorities.inc", but from what I can see. none of them work (you can try for yourself at "/cgi-bin/koha/opac-authoritiesdetail.pl") . I also did heaps of tests myself. If you PROCESS or INCLUDE another file, you get everything EXCEPT the BLOCK. It does seem like there are some ways around this though. 1) If you use the EXPOSE_BLOCKS option in the Template->new() constructor, you can directly INCLUDE/PROCESS a BLOCK from an included file. (http://mail.template-toolkit.org/pipermail/templates/2002-July/003460.html) . However, this option isn't documented in the official website at all, so I'd be hesitant to use this. 2) When you INCLUDE/PROCESS an external file, you can pass variables to it. You could pass a variable to the external file, and have it execute PROCESS/INCLUDE from within that file. At the moment, I'm doing the second. David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007

3 3

Koha 3.16.7 Release
by Mason James 26 Jan '15

26 Jan '15

The Koha release team is proud to announce the release of Koha 3.16.7. This is the latest release of the 3.16.x branch of Koha, this is a security and bugfix release. We strongly recommend all 3.16.x users upgrade to this latest version. Extra thanks in this release goes to Carlos Córdova, Larry Letelie, and BSZ-BW for reporting the security issues. As always you can wait for a few hours for the packages to appear at debian.koha-community.org or if you are still running from tarballs you can download them at http://downloads.koha-community.org You can read the full release notes at http://koha-community.org/koha-3-16-7-release Mason

1 0

The meta-packages koha-perldeps
by Zeno Tajoli 26 Jan '15

26 Jan '15

Hi Fridolin, Il 23/01/2015 14:10, Fridolin SOMERS ha scritto: > "koha-perldeps" is the name of the meta-package having all perl libs has > dependancies. > You can install it even if you do not install "koha-common". > > We use it to maintain our dev env. attention that the meta-package koha-perldeps (testd from stable and from dev) doesn't install ALL package that Koha needs. I find that same packages are not installed but you need them to use koha. The list is the same from stable and unstable repo. The list: Apache packages: apache2 apache2-mpm-itk apache2-utils apache2.2-bin apache2.2-common Yaz, Zebra, git and others packages: autopoint docbook-xml docbook-xsl gettext git git-man idzebra-2.0 idzebra-2.0-common idzebra-2.0-doc idzebra-2.0-utils javascript-common libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libcroco3 libcurl3-gnutls liberror-perl libgettextpo0 libicu-dev libidzebra-2.0 libidzebra-2.0-dev libidzebra-2.0-mod-alvis libidzebra-2.0-mod-dom libidzebra-2.0-mod-grs-marc libidzebra-2.0-mod-grs-regx libidzebra-2.0-mod-grs-xml libidzebra-2.0-mod-text libidzebra-2.0-modules libjs-jquery libjs-yui librtmp0 libssh2-1 libunistring0 libxml2-dev libxml2-utils libxslt1-dev libyaz5-dev perl-doc pwgen rsync sgml-data ssl-cert tcl8.4 tinymce unzip wwwconfig-common xmlstarlet xsltproc yaz So (for everyone) if you want to install ONLY koha packages from a Net install of Debian stable you need: -- From 'root' apt-get build-essential apt-get install systemd apt-get install sudo Insert 'koha' in sudoers Install koha repo I suggest to install also IndexData repo -- From 'koha' sudo apt-get update sudo apt-get upgrade sudo apt-get install koha-perldeps sudo apt-get install mysql-server [if you want mysql on the same hw of Koha] sudo apt-get install <the list above> To finish now those package must be installed/updated with CPAN: HTTPD::Bench::ApacheBench Test::DBIx::Class Archive::Extract DBIx::Class::Schema::Loader Test::WWW::Mechanize Except Archive::Extract (for Plugins) the others are mandatory only for development. Attention, this is the situation as I know it, do your tests if you want to be sure. Bye Zeno Tajoli -- Dr. Zeno Tajoli Soluzioni per la Ricerca Istituzionale - Automazione Biblioteche z.tajoli(a)cineca.it fax +39 02 2135520 CINECA - Sede operativa di Segrate

2 1