- Koha-devel - lists.koha-community.org

More info about Zebra hyphen truncating search (and announcing Zebra 2.0.60)
by David Cook 09 Feb '15

09 Feb '15

Hi all: Just letting everyone know that there is definitely a bug in Zebra 2.0.59 (and a certain number of unknown versions after 2.0.44 which is the version in Debian and Ubuntu prior to Jessie/Vivid which both use 2.0.59 as of this email), which causes problems with tokenization when using ICU and having a hyphen in a search term. Basically, the hyphen triggers tokenization, but only the first token is used. So a search for "Mont-Royal" will actually just be a search for "Mont". Or a search for "up-to-date" will just be a search for "up". This is the case even when trying to use ICU transformation/transliteration rules to remove the hyphen before tokenizing. However, I reported the issue to IndexData on February 4th, and they wrote a fix on February 7th (http://git.indexdata.com/?p=idzebra.git;a=commitdiff;h=704fd190292cb771df94 553b0ed6f9f4b71660a6). They've released that fix as part of Zebra 2.0.60, which is now available via the IndexData repositories. I just tested it on a Debian Squeeze install (using the IndexData apt repository), and it works great. I'm going to ask IndexData if they can provide more information, as Robin has volunteered to report a bug to Debian. It might be too late to get Zebra 2.0.60 shipped with Jessie, but maybe they can backport the patch or at least be aware of the issue... Cheers, David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007

1 0

Koha Community Newsletter: January 2015
by kohanews 06 Feb '15

06 Feb '15

Fellow Koha users: The Koha Community Newsletter for January 2015 is here: http://koha-community.org/koha-community-newsletter-january-2015/ Many thanks to the folks who submitted articles and news to this month's newsletter. Please feel free to email us with any corrections or suggestions. Thanks! Chad Roseburg Joanne Dillon Editors, Koha Community Newsletter

1 0

Re: [Koha-devel] Search speed
by Paul A 06 Feb '15

06 Feb '15

At 07:16 PM 2/4/2015 -0300, Tomas Cohen Arazi wrote: >For a fair comparison you should set facetNumRecs = 20 in >zebradb/zebra-biblios-dom.xml. Because otherwise use_zebra_facets=0 is >only processing the first 20 records, against 1000 from use_zebra_facets=1. Thanks for the suggestion. All tests stop/start memcached and zebrasrv. NYTProf gives: *** Both set to 20: Profile of ./opac-search.pl for 21.1s (of 21.3s), executing 1018674 statements and 252207 subroutine calls in 590 source files and 132 string evals. with 19.3s spent in Net::Z3950::ZOOM::resultset_record (xsub) and the CPU (from 'top') is stuck at 100% of a single core for all that time *** Set 20 (sysprefs) 1000 zebra-biblios-dom.xml: Profile of ./opac-search.pl for 20.9s (of 21.1s), executing 1018683 statements and 252793 subroutine calls in 590 source files and 133 string evals. with 19.1s spent in Net::Z3950::ZOOM::resultset_record (xsub) and the CPU at 100% *** Set 20 (sysprefs) 30000 zebra-biblios-dom.xml: Profile of ./opac-search.pl for 20.9s (of 21.2s), executing 1018669 statements and 252616 subroutine calls in 590 source files and 134 string evals. with 19.2s spent in Net::Z3950::ZOOM::resultset_record (xsub) and the CPU at 100% But (catastrophe), if I set both for 100 facets (maybe not unreasonable): Profile of ./opac-search.pl for 67.9s (of 68.2s), executing 1023091 statements and 255995 subroutine calls in 590 source files and 132 string evals, 66.2 secs in resultset_record with CPU 100% for that time. Conclusion: It's the syspref 'FacetMaxCount' that is the controlling value. >Regarding the amount of statements/function calls, I don't think they mean >too much. Some things got pulled from a +150 line piece of code into >several helper functions. I don't think it introduces much overhead. Maybe not, but several hundred thousand operations, even at a low median of 240 microsecs, adds up to 3.08 responding in 0.6 - 0.8 secs, and 3.18 (use_zebra_facets false) needing 1.3 to 1.6 And there are other aspects of "speed" that I'm looking at -- e.g. apache 2.4 with mod_deflate needs some fine tuning; otherwise images, css, js (maybe more) don't necessarily get compressed and add considerably to the number of packets, syns, acks, etc and web latency can kick in. But that's for another day... ;=) Again thanks for the suggestion, best -- Paul >REgards >Tomas > >PS. if you play with facetNumRecs, remember to restart zebra each time you >change it. > >On Wed, Feb 4, 2015 at 6:22 PM, Paul A <paul.a(a)navalmarinearchive.com> wrote: >At 02:02 PM 2/2/2015 +0000, Tomas Cohen Arazi wrote: >Paul, while you are running your tests, could you please compare this both >scenarios: >- <use_zebra_facets>1</use_zebra_facets> >- <use_zebra_facets>0</use_zebra_facets> >Thanks >PS: I have found that searchResults is too slow too. > > >I have posted notes, results and some NYTProf examples at ><http://navalmarinearchive.com/z_koha/> -- the results are repeatable (I >must have run NYTProf nearly 200 times.) > >Provisional conclusions are that <use_zebra_facets> = 1 (true) is not >usable in production for a search that finds > 20k results. For smaller >results, it might be acceptable, but is still appreciably slower than >3.08. Under the best circumstances (0-5 results) 3.08 responds in < 0.7 >secs, 3.18 in 1.3 secs.Â At the other extreme (25k results), 3.08 >requires 1.07 secs, 3.18 21+ secs. > >I have defined an "optimal" 3.18 combination (see green line, Graph 2) >which runs at about half the speed of 3.08, but does not give 'Series', >'Places' and 'Titles' in the facets. > >One point that jumped out is that a best case 3.08 search executes 431207 >statements and 76556 subroutine calls in 297 source files and 81 string >evals whereas a worst case 3.18 executes 1238167 statements and 309905 >subroutine calls in 591 source files and 132 string evals. (Even the worst >case 308 appears much more efficient than the best case 3.18.) > >The use of -T to start Zebra is non-conclusive -- it vaguely slows down >smaller result sets and vaguely speeds up larger result sets (testing is >repeatable.)Â Robin was of course quite correct in his recent email, but >I cannot explain even the minor variations experienced. > >I'm not at all certain how to file this as a bug, so will await any comments. > >Best -- Paul

1 0

A report for finding records with multiple items?
by andrea.furin＠slacky.it 06 Feb '15

06 Feb '15

Hi. I'd need to write a report for finding records with multiple items in them: any suggestions? TIA Andrea Furin

2 1

Dev meeting part 2 missed
by Tomas Cohen Arazi 06 Feb '15

06 Feb '15

Hi, I got home late and it seems there hasn't been part 2 of the dev meeting. People that didn't attend part 1, please propose a date/time. Important votes regarding stuff removal are being held. -- Tomás Cohen Arazi Prosecretaría de Informática Universidad Nacional de Córdoba ✆ +54 351 5353750 ext 13168 GPG: B76C 6E7C 2D80 551A C765 E225 0A27 2EA1 B2F3 C15F

2 1

Marc in spreadsheet
by andrea.furin＠slacky.it 05 Feb '15

05 Feb '15

Hi. I need a way to transform my marc file from koha in a spreadsheet file (excel or libreoffice): there is a way? Thanks in advance Andrea Furin

1 0

Re: [Koha-devel] Search speed
by Paul A 05 Feb '15

05 Feb '15

At 10:03 AM 2/5/2015 +0100, Jonathan Druart wrote: >I have filled bug 13665 yesterday >http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13665 Thanks. I had come to that same conclusion concerning resultset_record (see my email 9 Jan to this list.) For anyone wanting to start chasing this down, I noted: ZOOM modules: We have Net::Z3950::ZOOM (v1.26) Ubuntu libnet-z3950-zoom-perl (latest.) Includes libidzebra-2.0-mod-alvis, but there are two different files named ZOOM.pm me@hood:/$ cat -n /usr/lib/perl5/ZOOM.pm 1110 lines me@hood:/$ cat -n /usr/lib/perl5/Net/Z3950/ZOOM.pm 194 lines Debug: "book mark" 'f ZOOM' both choose the short one: 'b 152' therefore force 'f /usr/lib/perl5/ZOOM.pm' 'b 739' 'f /usr/share/koha/lib/C4/Auth.pm' 'b 442' return to 'f opac-search.pl' But from then on, chasing individual lines/functions (s, n, r) became frustrating, and a dump (T) was unmanageable. Again, thanks -- Paul >2015-02-04 22:22 GMT+01:00 Paul A <paul.a(a)navalmarinearchive.com>: > > At 02:02 PM 2/2/2015 +0000, Tomas Cohen Arazi wrote: > >> > >> Paul, while you are running your tests, could you please compare this both > >> scenarios: > >> - <use_zebra_facets>1</use_zebra_facets> > >> - <use_zebra_facets>0</use_zebra_facets> > >> Thanks > >> PS: I have found that searchResults is too slow too. > > > > > > I have posted notes, results and some NYTProf examples at > > <http://navalmarinearchive.com/z_koha/> -- the results are repeatable (I > > must have run NYTProf nearly 200 times.) > > > > Provisional conclusions are that <use_zebra_facets> = 1 (true) is not > usable > > in production for a search that finds > 20k results. For smaller > results, it > > might be acceptable, but is still appreciably slower than 3.08. Under the > > best circumstances (0-5 results) 3.08 responds in < 0.7 secs, 3.18 in 1.3 > > secs. At the other extreme (25k results), 3.08 requires 1.07 secs, > 3.18 21+ > > secs. > > > > I have defined an "optimal" 3.18 combination (see green line, Graph 2) > which > > runs at about half the speed of 3.08, but does not give 'Series', 'Places' > > and 'Titles' in the facets. > > > > One point that jumped out is that a best case 3.08 search executes 431207 > > statements and 76556 subroutine calls in 297 source files and 81 string > > evals whereas a worst case 3.18 executes 1238167 statements and 309905 > > subroutine calls in 591 source files and 132 string evals. (Even the worst > > case 308 appears much more efficient than the best case 3.18.) > > > > The use of -T to start Zebra is non-conclusive -- it vaguely slows down > > smaller result sets and vaguely speeds up larger result sets (testing is > > repeatable.) Robin was of course quite correct in his recent email, but I > > cannot explain even the minor variations experienced. > > > > I'm not at all certain how to file this as a bug, so will await any > > comments. > > > > Best -- Paul > > > > > > _______________________________________________ > > Koha-devel mailing list > > Koha-devel(a)lists.koha-community.org > > http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel > > website : http://www.koha-community.org/ > > git : http://git.koha-community.org/ > > bugs : http://bugs.koha-community.org/ --- Maritime heritage and history, preservation and conservation, research and education through the written word and the arts. <http://NavalMarineArchive.com> and <http://UltraMarine.ca>

1 0

Re: [Koha-devel] Search speed
by Paul A 05 Feb '15

05 Feb '15

At 02:02 PM 2/2/2015 +0000, Tomas Cohen Arazi wrote: >Paul, while you are running your tests, could you please compare this both >scenarios: >- <use_zebra_facets>1</use_zebra_facets> >- <use_zebra_facets>0</use_zebra_facets> >Thanks >PS: I have found that searchResults is too slow too. I have posted notes, results and some NYTProf examples at <http://navalmarinearchive.com/z_koha/> -- the results are repeatable (I must have run NYTProf nearly 200 times.) Provisional conclusions are that <use_zebra_facets> = 1 (true) is not usable in production for a search that finds > 20k results. For smaller results, it might be acceptable, but is still appreciably slower than 3.08. Under the best circumstances (0-5 results) 3.08 responds in < 0.7 secs, 3.18 in 1.3 secs. At the other extreme (25k results), 3.08 requires 1.07 secs, 3.18 21+ secs. I have defined an "optimal" 3.18 combination (see green line, Graph 2) which runs at about half the speed of 3.08, but does not give 'Series', 'Places' and 'Titles' in the facets. One point that jumped out is that a best case 3.08 search executes 431207 statements and 76556 subroutine calls in 297 source files and 81 string evals whereas a worst case 3.18 executes 1238167 statements and 309905 subroutine calls in 591 source files and 132 string evals. (Even the worst case 308 appears much more efficient than the best case 3.18.) The use of -T to start Zebra is non-conclusive -- it vaguely slows down smaller result sets and vaguely speeds up larger result sets (testing is repeatable.) Robin was of course quite correct in his recent email, but I cannot explain even the minor variations experienced. I'm not at all certain how to file this as a bug, so will await any comments. Best -- Paul

3 2

Zebra hyphen truncating search
by David Cook 05 Feb '15

05 Feb '15

Hi Colin! I stumbled across an email you sent to the Indexdata Zebralist in 2013: http://lists.indexdata.dk/pipermail/zebralist/2013-August/002576.html Were you ever able to solve those problems? Also, what version of Zebra were you running? I've noticed this problem with Zebra using 2.0.59, but I haven't been able to produce it using Zebra 2.0.47. I had the exact same configuration files, MySQL database, and Zebra indexes. Here's an example: 2.0.59: Z> f UPA-CE14060101 Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 12325, setno 1 SearchResult-1: term=UPA cnt=12325 records returned: 0 Elapsed: 0.021128 2.0.47: Z> f UPA-CE14060101 Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 1, setno 3 SearchResult-1: term=UPA cnt=12325, term=CE14060101 cnt=1 records returned: 0 Elapsed: 0.015657 -- It looks like the packages in Debian 7 and Ubuntu 14.04 are stuck back at Zebra 2.0.44, which is from 2010. However, Indexdata apparently has newer deb packages for both Debian and Ubuntu: http://ftp.indexdata.dk/pub/zebra/. In fact, the server I'm having problems with appears to have installed Zebra 2.0.59 using the following "/etc/apt/sources.list.d/indexdata.list" pointing to "deb http://ftp.indexdata.dk/debian squeeze main". I'm guessing that you were using a Zebra version that was higher than 2.0.47? Possibly a deb package installed from Indexdata's repos? I know Tomas has had good luck contacting Indexdata folk. maybe he has some ideas about how to find a solution to this issue? David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007

3 4

Re: [Koha-devel] Zebra hyphen truncating search Part Deux
by David Cook 04 Feb '15

04 Feb '15

Robin just pointed out that Debian Jessie (8) will be switching to 2.0.59 and I noticed that Ubuntu Vivid (15.04) will as well. That means that Koha package installs that use ICU on these newer OSes will start running into this problem as well. (Actually, most newer OSes will have this issue. OpenSuse 13.2 released in November 2014 is using 2.0.59.) So... that's not good. We first ran into this problem because of a barcode search "ABC-12345" was returning all barcodes that started with "ABC". However, you'll run into this problem for a lot of keyword and title searches. For instance "Mont-Royal" will return all records containing "Mont". Note that this is only for ICU. If you're using CHR, "Mont-Royal" will just get you records that have "Mont" and "Royal" in them (since Zebra/YAZ breaks on hyphens when using CHR character mapping). Anyway, I've sent an email to Indexdata, so I'm going to try pursuing this with them as much as possible... David Cook Systems Librarian Prosentient Systems 72/330 Wattle St, Ultimo, NSW 2007 > -----Original Message----- > From: Colin Campbell [mailto:colin.campbell@ptfs-europe.com] > Sent: Tuesday, 3 February 2015 8:03 PM > To: David Cook > Cc: 'Colin Campbell'; koha-devel(a)lists.koha-community.org > Subject: Re: Zebra hyphen truncating search > > On Tue, Feb 03, 2015 at 05:16:05PM +1100, David Cook wrote: > > Hi Colin! > > > > > > > > I stumbled across an email you sent to the Indexdata Zebralist in 2013: > > http://lists.indexdata.dk/pipermail/zebralist/2013-August/002576.html > > > > > > > > Were you ever able to solve those problems? > > I think its still the case. It can be further complicated by what the settings in > the system are and how Search.pm interprets them in constructing the > search. Especially the relevancy options. > > > > Also, what version of Zebra were you running? I've noticed this > > problem with Zebra using 2.0.59, but I haven't been able to produce it > > using Zebra 2.0.47. I had the exact same configuration files, MySQL > > database, and Zebra indexes. Here's an example: > It would be the current releases from indexdata on Debian and also built > from source on Fedora. The indexdata debian packages track the source > closely. > > Cheers > Colin > > -- > Colin Campbell > Chief Software Engineer, > PTFS Europe Limited > Content Management and Library Solutions > +44 (0) 800 756 6803 (phone) > +44 (0) 7759 633626 (mobile) > colin.campbell(a)ptfs-europe.com > skype: colin_campbell2 > > http://www.ptfs-europe.com

1 0