http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #3 from Robin Sheat <robin@catalyst.net.nz> --- Created attachment 29204 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=29204&action=edit Bug 12478 - an almagamation of all the Elasticsearch code so far -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #5 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Robin Sheat from comment #4)

<elasticsearch> <server>es-server:9200</server>

<server> can be repeated to point to each server in your cluster, too. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Magnus Enger <magnus@enger.priv.no> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |magnus@enger.priv.no -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Zeno Tajoli <z.tajoli@cineca.it> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |z.tajoli@cineca.it -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Dobrica Pavlinusic <dpavlin@rot13.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dpavlin@rot13.org -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #8 from Robin Sheat <robin@catalyst.net.nz> --- Created attachment 31129 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=31129&action=edit Bug 12478 - set up database tables for elasticsearch This sets up the database changes needed to run ES. This isn't intended to be ready for upstreaming, just to make testing easier. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #10 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to claire.hernandez@biblibre.com from comment #9)

Hi, is there somewhere a target described with features that could be provided ? like a rfc or summary or something like that ?

There isn't really, at this stage. For now it's still a case of replicating the functionality of zebra. Once that is done, then new features can be added. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Robin Sheat <robin@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #31128|0 |1 is obsolete| | Attachment #31129|0 |1 is obsolete| | --- Comment #12 from Robin Sheat <robin@catalyst.net.nz> --- Created attachment 33136 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=33136&action=edit Bug 12478 - add some base objects that the ES code will depend on -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #14 from Robin Sheat <robin@catalyst.net.nz> --- Created attachment 33138 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=33138&action=edit Bug 12478 - pile of elasticsearch code -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #16 from Robin Sheat <robin@catalyst.net.nz> --- Created attachment 33140 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=33140&action=edit Bug 12478 - authorities can now be stored in ES (Not fetched yet though.) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #18 from Robin Sheat <robin@catalyst.net.nz> --- I've added a dump of the current state of patches. It's in the process of being split up and having unit tests written, in particular of the underlying modules, i.e. the things that will be needed but aren't central to ES itself. The last functional change was the ability for authorities to be indexed too. Currently, they can't be searched, but they are there. The next functional change (clearly) is to make them be searchable. I'm expecting that 90% of this will leverage the existing query builder type stuff. I'm hoping that someone can have a go setting this up on their own installation and trying it out, to see if there are any particular points that need explained. dcook, I'll be seeing you at the conference tomorrow, I suggest you have a laptop with a VM ready to go on it ;) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Indranil Das Gupta <indradg@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |indradg@gmail.com -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Jonathan Druart <jonathan.druart@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@biblibre.co | |m --- Comment #21 from Jonathan Druart <jonathan.druart@biblibre.com> --- Is there a doc somewhere? I did not find it... I tried but failed: $ git remote add catalyst git://git.catalyst.net.nz/koha.git $ git checkout -b elastic_search catalyst/elastic_search $ perl installer/data/mysql/updatedatabase.pl $ sudo apt-get install elasticsearch $ perl misc/search_tools/rebuild_elastic_search.pl -h Can't locate Elasticsearch.pm in @INC (you may need to install the Elasticsearch module) (@INC contains: blablabla) $ apt-cache search elasticsearch | grep perl libcatmandu-perl - metadata toolkit $ sudo apt-get install libcatmandu-perl But got the same error. The ElasticSearch module on the cpan is marked as deprecated: http://search.cpan.org/~drtech/ElasticSearch-0.68/lib/ElasticSearch.pm -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #23 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Jonathan Druart from comment #21)

$ perl misc/search_tools/rebuild_elastic_search.pl -h Can't locate Elasticsearch.pm in @INC (you may need to install the Elasticsearch module) (@INC contains: blablabla)

$ apt-cache search elasticsearch | grep perl libcatmandu-perl - metadata toolkit

In comment #6 I added a link to the dependencies needed. It's a bit old now, I've been working on making some new ones, but that archive may still work. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #25 from Robin Sheat <robin@catalyst.net.nz> --- Yeah, they worked on Ubuntu 14.10 last I looked. I'm in the process of making a new set for Debian Jessie. As for the deprecated bits, there are some issues there that make it not quite worth messing with the new versions just yet. Check out the discussion thread here: http://mail.librecat.org/pipermail/librecat-dev/2015-January/000322.html There should be very few, if any, API changes as the result of switching to the new version when it's suitable, too. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #27 from Jonathan Druart <jonathan.druart@biblibre.com> --- Looks better with these new packages. Continuing debugging... Catmandu::Importer::MARC was missing, I installed it using cpanm: $ sudo cpanm Catmandu::Importer::MARC Now I got: $ perl rebuild_elastic_search.pl 1 to 115 is displayed Committing... And nothing else... $ ps aux | grep elasticsearch returns nothing $ sudo service elasticsearch start $ ps aux | grep elasticsearch returns nothing $ curl -X GET http://localhost:9200/ curl: (7) Failed to connect to localhost port 9200: Connection refused Editing /etc/init.d/elasticsearch, I found the binary: /usr/share/elasticsearch/bin/elasticsearch $ cd /usr/share/elasticsearch $ slog4j:WARN No appenders could be found for logger (node). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.udo bin/elasticsearch $ apt-cache show elasticsearch Package: elasticsearch Version: 1.0.3+dfsg-5 Should I installed another version? Something is missing in my jvm configuration? Something to do in the elasticsearch config? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #29 from Jonathan Druart <jonathan.druart@biblibre.com> --- $ sudo apt-get install libcatmandu-marc-perl Try with the lastest (1.4.2): $ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch... $ sudo dpkg -i elasticsearch-1.4.2.deb Selecting previously unselected package elasticsearch. (Reading database ... 59178 files and directories currently installed.) Preparing to unpack .../koha/elasticsearch-1.4.2.deb ... Unpacking elasticsearch (1.4.2) ... Setting up elasticsearch (1.4.2) ... Adding system user `elasticsearch' (UID 106) ... Adding new user `elasticsearch' (UID 106) with group `elasticsearch' ... Not creating home directory `/usr/share/elasticsearch'. ### NOT starting elasticsearch by default on bootup, please execute sudo update-rc.d elasticsearch defaults 95 10 ### In order to start elasticsearch, execute sudo /etc/init.d/elasticsearch start $ sudo update-rc.d elasticsearch defaults 95 10 $ sudo /etc/init.d/elasticsearch start $ ps aux | grep elasticsearch elastic+ 5715 75.4 1.6 1326700 131172 ? Sl 09:57 0:03 /usr/lib/jvm/java-7-openjdk-i386//bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.4.2.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch better :) But the indexing is still stuck on "Committing..." I am using an UNIMARC DB, does the problem could be related to a bad mapping? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |katrin.fischer@bsz-bw.de --- Comment #31 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Robin, having a discussion about his on IRC right now - we are all quite curious about your work - could you write up some installation docs and/or a summary what the ES patches will include/not include? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #32 from Jonathan Druart <jonathan.druart@biblibre.com> --- (In reply to Robin Sheat from comment #30)

(In reply to Jonathan Druart from comment #29)

...

Try with the lastest (1.4.2):

$ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch... 1.4.2.deb

For what it's worth, I'm using 1.3.7 which is the latest stable. I wouldn't think that would make a difference really though.

Will try with 1.3.7, but I believe I got the same error as before ( with this version.

Sitting on "committing" is consistent with not being able to talk to the ES server, and there's currently no error handling for that (I'd expect it to time out eventually, but who knows how long that'll take.) If I were you, that's where I'd start looking. Make sure you can hit it with curl, and that the config in koha-conf.xml is correct. fwiw, mine is:

<elasticsearch> <server>koha-es:9200</server> <index_name>koha_robin</index_name> </elasticsearch>

Yes, I have something similar: 303 <elasticsearch> 304 <server>http://localhost:9200/</server> 305 <index_name>koha_instance</index_name> 306 </elasticsearch>

If you're keen, fire it up in the debugger (perl -d) and do:

c Catmandu::Store::ElasticSearch::Bag::commit

Actually it never reach commit: % perl -d misc/search_tools/rebuild_elastic_search.pl Loading DB routines from perl5db.pl version 1.44 Editor support available. Enter h or 'h h' for help, or 'man perldebug' for more help. main::(misc/search_tools/rebuild_elastic_search.pl:92): 92: my $verbose = 0; DB<1> c Catmandu::Addable::add_many Indexing biblios 3 Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:84): 84: my ($self, $many) = @_; DB<2> n Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:86): 86: if (is_hash_ref($many)) { DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:91): 91: if (is_array_ref($many)) { DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:96): 96: if (is_invocant($many)) { DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:97): 97: $many = check_able($many, 'generator')->generator; DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:100): 100: check_code_ref($many); DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:102): 102: my $data; DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:103): 103: my $n = 0; DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:104): 104: while (defined($data = $many->())) { DB<2> use Data::Dumper; warn Dumper $many; $VAR1 = sub { "DUMMY" }; at (eval 1256)[/usr/share/perl/5.20/perl5db.pl:732] line 2. eval 'no strict; ($@, $!, $^E, $,, $/, $\\, $^W) = @DB::saved;package Catmandu::Addable; $^D = $^D | $DB::db_stop; use Data::Dumper; warn Dumper $many;; ' called at /usr/share/perl/5.20/perl5db.pl line 732 DB::eval called at /usr/share/perl/5.20/perl5db.pl line 3094 DB::DB called at /usr/share/perl5/Catmandu/Addable.pm line 104 Catmandu::Addable::add_many(Catmandu::Store::ElasticSearch::Bag=HASH(0xd5c503c), Catmandu::Iterator=CODE(0xc565080)) called at Koha/ElasticSearch/Indexer.pm line 80 Koha::ElasticSearch::Indexer::update_index(Koha::ElasticSearch::Indexer=HASH(0xcd60c00), ARRAY(0xca8ca80), ARRAY(0xca8cae4)) called at misc/search_tools/rebuild_elastic_search.pl line 184 main::do_reindex(CODE(0xcae4dc0), "biblios") called at misc/search_tools/rebuild_elastic_search.pl line 132 DB<3> warn ref($many); CODE at (eval 1257)[/usr/share/perl/5.20/perl5db.pl:732] line 2. at (eval 1257)[/usr/share/perl/5.20/perl5db.pl:732] line 2. eval 'no strict; ($@, $!, $^E, $,, $/, $\\, $^W) = @DB::saved;package Catmandu::Addable; $^D = $^D | $DB::db_stop; warn ref($many);; ' called at /usr/share/perl/5.20/perl5db.pl line 732 DB::eval called at /usr/share/perl/5.20/perl5db.pl line 3094 DB::DB called at /usr/share/perl5/Catmandu/Addable.pm line 104 Catmandu::Addable::add_many(Catmandu::Store::ElasticSearch::Bag=HASH(0xd5c503c), Catmandu::Iterator=CODE(0xc565080)) called at Koha/ElasticSearch/Indexer.pm line 80 Koha::ElasticSearch::Indexer::update_index(Koha::ElasticSearch::Indexer=HASH(0xcd60c00), ARRAY(0xca8ca80), ARRAY(0xca8cae4)) called at misc/search_tools/rebuild_elastic_search.pl line 184 main::do_reindex(CODE(0xcae4dc0), "biblios") called at misc/search_tools/rebuild_elastic_search.pl line 132 DB<4> n And nothing else (no CPU activity neither). the ref($many) returning "CODE" does not smell good :) continuing DB<4> n ^CqMARC::File::USMARC::_next(/usr/share/perl5/MARC/File/USMARC.pm:53): 53: local $/ = END_OF_RECORD; DB<4> q % pmvers MARC::File::USMARC /usr/bin/pmvers: unknown version for module `MARC::File::USMARC' hum... Since I installed it using the CPAN first, I removed it: % sudo cpanm -U Catmandu::Importer::MARC [...] Successfully uninstalled Catmandu::Importer::MARC % dpkg -l libcatmandu-marc-perl ii libcatmandu-marc-perl | 0.206-2 | all | modules for working with MARC data within the Catmandu framework % perl misc/search_tools/rebuild_elastic_search.pl [lot of logs] 25888 records indexed. \o/ Sorry about that! I should have tried pmpath MARC::File::USMARC to catch the problem quickly. Now, search records! OPAC: cgi-bin/koha/opac-search.pl returns Can't locate Koha/ElasticSearch/Search.pm in @INC It's seem to be caused by the last commit Bug 12478 - authority work in progress diff --git a/Koha/ElasticSearch/Search.pm b/Koha/ElasticSearch/Search.pm deleted file mode 100644 I tried to fix some compilation errors (patch coming), but the branch looks to be let in an unusable state. The next error is: Can't locate object method "mk_accessors" via package "Koha::SearchEngine::ElasticSearch::Search" I don't want to continue and add conflicts with something you have already fixed. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #34 from Jonathan Druart <jonathan.druart@biblibre.com> --- Created attachment 35654 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=35654&action=edit Bug 12478: fix some compilation errors Feel free to obsolete this patch. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #35 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Jonathan Druart from comment #33)

previous comment tldr: indexing: OK

Yay!

searching: KO, branch in an unusable state.

Yeah, I forgot to mention that sorry. Roll it back a bit, I'm in the middle of a bit of refactoring (just moving a module to a more consistent place, but I haven't updated the references to it yet), and also developing the authorities searching (though only some of that has been pushed so far.) I'll try to get it to a properly working state again today. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Robin Sheat <robin@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #35654|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #38 from Jonathan Druart <jonathan.druart@biblibre.com> --- Thanks Robin for the wiki page. I have quickly tested this morning, to confirm the search works. Just some quick notes: - The indexation should commit every 1k biblio (minimum), the commit is a slow op indeed. - You have removed the use of search/results.tt in your last commit, is it intended? I am pretty sure it's not a good idea to use the same template for ES and Zebra, but maybe it's temporary. Note that there are encoding errors in facets (and in the table results also), the "Show more" link appears even if only 1 entry is displayed. I only get 2 facets: authors and itemtype. Do you plan to rebase your work against master? It would be great to see these patches on top of bug 11944. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #40 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Robin Sheat from comment #39)

MARC21 or UNIMARC? You should get more, though I haven't looked at that for a while.

For example, http://elasticsearch.koha.catalystdemo.net.nz/cgi-bin/koha/opac-search. pl?q=chicken

give availability, item types, authors and topics. But the facet stuff as a whole will need more work, I mostly got it to the point where it works at all and then moved on.

Oh, whether a field is suitable for faceting is now defined in the mapping table in the database. This might be why it's not working if you're in unimarc, the field definitions may not be correct for it. I want (eventually) to extend this to make what is facetable configurable, at the moment it's hardcoded into the template. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Peter Zhao <peterzhaonj@163.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |peterzhaonj@163.com --- Comment #42 from Peter Zhao <peterzhaonj@163.com> --- (In reply to Robin Sheat from comment #41)

FYI, the catalyst repo branch now has basic authority search working. There's still a good bit to do (paging, biblio counts, many more things aren't there yet), but results are coming out so I'm counting that as a win :)

You can see it in action here:

http://elasticsearch.koha.catalystdemo.net.nz/cgi-bin/koha/opac-authorities- home. pl?op=do_search&type=opac&operator=contains&value=robert&marclist=any&and_or= and&orderby=HeadingAsc

I think the fact that there are so many Jordan, Roberts is due to the data, but I haven't actually checked yet. At the moment it just replicates how the zebra version works, but I do want to push some of the things into the indexing side so that there's less computation needed to display results and things can perhaps be made a bit simpler. We can do this because we can store arbitrary fields in elasticsearch alongside the actual records.

Dear Robin, It is a great job! Thanks a lot. I tried to install a ES. It seems to index the biblio. But I can't search the record."No results found!" I can find the record by Zebra. koha@koha:~$ /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl -v -d Indexing biblios 1 Could you give some advice? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #43 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Peter Zhao from comment #42)

It is a great job! Thanks a lot. I tried to install a ES. It seems to index the biblio. But I can't search the record."No results found!" I can find the record by Zebra. koha@koha:~$ /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl -v -d Indexing biblios 1

Could you give some advice?

There's not really enough info to go on there. I'd start by looking in the logs. Many operations should be filling them full of search traces and so on, so that might have useful things in it. Have you followed the steps in here: http://wiki.koha-community.org/wiki/Elasticsearch in particular, adding the mapping SQL file into the database. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #45 from Peter Zhao <peterzhaonj@163.com> --- (In reply to Robin Sheat from comment #43)

(In reply to Peter Zhao from comment #42)

...

It is a great job! Thanks a lot. I tried to install a ES. It seems to index the biblio. But I can't search the record."No results found!" I can find the record by Zebra. koha@koha:~$ /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl -v -d Indexing biblios 1

Could you give some advice?

There's not really enough info to go on there. I'd start by looking in the logs. Many operations should be filling them full of search traces and so on, so that might have useful things in it.

Have you followed the steps in here: http://wiki.koha-community.org/wiki/Elasticsearch in particular, adding the mapping SQL file into the database.

The following is the "koha-opac-error log " [Fri Mar 06 12:30:19.551872 2015] [cgi:error] [pid 3674] [client 127.0.0.1:45890] AH01215: [Fri Mar 6 12:30:19 2015] opac-search.pl: Use of uninitialized value $f in hash element at /usr/share/koha/lib/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 479., referer: http://127.0.1.1/cgi-bin/koha/opac-search.pl?idx=ti&q=theology [Fri Mar 06 12:33:43.232092 2015] [cgi:error] [pid 3759] [client 127.0.0.1:45904] AH01215: [Fri Mar 6 12:33:43 2015] opac-search.pl: Use of uninitialized value $f in hash element at /usr/share/koha/lib/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 479., referer: http://127.0.1.1/cgi-bin/koha/opac-search.pl?idx=&q=theology [Fri Mar 06 12:33:43.398916 2015] [cgi:error] [pid 3759] [client 127.0.0.1:45904] AH01215: [Fri Mar 6 12:33:43 2015] opac-search.pl: Use of uninitialized value $error in concatenation (.) or string at /usr/share/koha/opac/cgi-bin/opac/opac-search.pl line 568., referer: http://127.0.1.1/cgi-bin/koha/opac-search.pl?idx=&q=theology -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #47 from Peter Zhao <peterzhaonj@163.com> --- (In reply to Robin Sheat from comment #46)

It might be useful to ensure that they indexed right, you can do this:

curl -XGET 'http://localhost:9200/koha_biblios/_search?pretty=1'

where 'koha_biblios' is whatever your index is called. This'll give you a dump of everything in the index.

----------- $ curl -XGET 'http://localhost:9200/koha_biblios/_search?pretty=1' { "error" : "IndexMissingException[[koha_biblios] missing]", "status" : 404 } ---------- $ /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl -v -d Indexing biblios 1 ----------------- I just catalogued one record to try. But it always stay there. I think it cannot finish to build the index. How to handle this problem? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #49 from Robin Sheat <robin@catalyst.net.nz> --- Yay, the fundamentals of authority searching are now working (biblio count and paging.) Next up is making sure the various types of searches result in the right results. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #51 from Robin Sheat <robin@catalyst.net.nz> --- How long did you let it sit on 'Committing', the first one can take some time (in the order of 10 seconds or so perhaps.) Otherwise, it's hard to see what's going on without real information but it does look like things aren't indexing well. In Koha/ElasticSearch/Indexer.pm there are a couple of commented out 'trace_calls' entries. If you make that active, you may get more information about what's going on. Otherwise perhaps trace through it using the Perl debugger and see where it stops. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #53 from Peter Zhao <peterzhaonj@163.com> --- (In reply to Peter Zhao from comment #52)

(In reply to Robin Sheat from comment #51)

...

How long did you let it sit on 'Committing', the first one can take some time (in the order of 10 seconds or so perhaps.)

Otherwise, it's hard to see what's going on without real information but it does look like things aren't indexing well.

In Koha/ElasticSearch/Indexer.pm there are a couple of commented out 'trace_calls' entries. If you make that active, you may get more information about what's going on.

Otherwise perhaps trace through it using the Perl debugger and see where it stops.

~$ perl -d /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl

Loading DB routines from perl5db.pl version 1.33 Editor support available.

Enter h or `h h' for help, or `man perldebug' for more help.

main::(/home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl:92): 92: my $verbose = 0; DB<1>

The problem is fixed by "cpan Task::Catmandu". Index works well. ~$ /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl -v -d Indexing biblios 1 Committing... 1 records indexed. Indexing authorities 0 records indexed. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #55 from Robin Sheat <robin@catalyst.net.nz> --- Authority searching is now "complete", by which I mean it seems to work but there are bound to be issues in it. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #56 from Robin Sheat <robin@catalyst.net.nz> --- I've rebased the branch on top of current master* and pushed it into the catalyst repo. This may cause the SearchEngine preference to vanish, if so, just go and re-set it from the system preferences. * git can get upset if a file is in the thing you're rebasing onto, but there are still patches that have to go against it. It ended up grabbing a totally unrelated file that presumably was the closest match and shoving the patches in there. That's why there's the occasionally weird touching of Koha::Template::Plugin::Price. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Ere Maijala <ere.maijala@helsinki.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ere.maijala@helsinki.fi -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #58 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Robin Sheat from comment #57)

Just a note that I've got basic staff client working on my demo server: http://elasticsearch.koha.catalystdemo.net.nz/

Staff client /searching/ -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #59 from Jonathan Druart <jonathan.druart@biblibre.com> --- (In reply to Robin Sheat from comment #57)

Just a note that I've got basic staff client working on my demo server: http://elasticsearch.koha.catalystdemo.net.nz/

"Basic searching there should be working, anything else may explode or otherwise fail." Does it mean we can make you some feedbacks or you know where bugs are? :) (for instance a search result for "harry" sorted by author az is not the reverse of author za. The "more" link does not do anything.) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #61 from Robin Sheat <robin@catalyst.net.nz> --- Just a heads up that I've refactored how the mappings get stored in the database. If you have a local setup, you'll want to re-import the elasticsearch_mapping.sql (which currently is a weird hybrid, it loads things in the old way, uses SQL to generate new tables from them, and drops the old one. This is temporary until I get around to verifying that the new version is near enough to correct. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #63 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Juan Romay Sieira from comment #62)

I'm testing ES. My problem is that Koha is always trying to connect to localhost:9200, but my koha-conf.xml has:

<elasticsearch> <server>192.168.0.213:9200</server> <index_name>core</index_name> </elasticsearch>

my instance of ES it not in localhost. Do I have to do something to force to connect to 192.168.0.213 and not to localhost?

This is the error: [NoNodes] ** No nodes are available: [http://localhost:9200], called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/local/share/perl/5.10.1/Catmandu/Store/ElasticSearch.pm line 61

It definitely doesn't require localhost, my own test system runs against a remote ES server. I'm not sure what could cause that, are you sure you've got the right koha-conf? I'll attach the whole of mine to make sure the context is right. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #65 from Juan Romay Sieira <juan.sieira@xercode.es> --- Yes, its the right koha-conf, and Koha gets the right configuration. Now I change it to put the timeout tag and . I put a warn in Koha/SearchEngine/Elasticsearch/Search.pm warn Data::Dumper::Dumper(\%$params); $self->store( Catmandu::Store::ElasticSearch->new( %$params, trace_calls => 1, ) ) unless $self->store; this is what it show in koha opac log $VAR1 = { 'index_name' => 'core_biblios', 'timeout' => '10', 'servers' => [ '192.168.0.213:9200' ] }; The ES instance is running too, if I visit the URL of ES in a browser it returns: { "status" : 200, "name" : "koha-es", "cluster_name" : "koha-cluster", "version" : { "number" : "1.6.0", "build_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0", "build_timestamp" : "2015-06-09T13:36:34Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" } To temporaly fix it I wrote a iptables rule to forward my 9200 port to the other machine... -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #67 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Juan Romay Sieira from comment #66)

Finally I can do index, search, ... and with the same configuration. My problem is that I was using Squeeze instead Jessie. Thank you!

Oh, excellent! I'm surprised it even let you install the modules on squeeze, to be honest. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #69 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Hi Robin, thx for updating! Some things I tried/found/wondered about, mostly about facetting as I am getting a lot of feedback on our current implementation: 1) Can you show the number of results behind the facets? There is currently a preference for this, but I couldn't turn it on/chefk if it turned on. 2) Can facets be sorted by most used to less used in a result list? 3) Can you show all entries for a big result list? How does it limit which facets to show? 4) If you search for () it explodes rather spectacularly. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #70 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Katrin Fischer from comment #69)

1) Can you show the number of results behind the facets? There is currently a preference for this, but I couldn't turn it on/chefk if it turned on.

I've turned it on now.

2) Can facets be sorted by most used to less used in a result list?

This is a Koha question, not a search question. We can re-order them however we like (though, I don't know how meaningful that'd be tbh.) I'm trying to not change the behaviour significantly beyond what zebra can do.

3) Can you show all entries for a big result list? How does it limit which facets to show?

That's not how facets work in this case. It calculates facets across all the results (roughly, but I think mostly accurate for the scale of results in a Koha system.) It then returns the most frequent 10. At the moment, that's what's displaying. When I get to that point in my todo list, it'll show 5 until you click on the "show more" thing. Which forces a full page reload for hysterical raisins.

4) If you search for () it explodes rather spectacularly.

It sure does. There's an option to make the string parsing more lenient I need to enable. I also need to catch that error better. Basically, if you give it anything that can't be parsed as a properly lucene query, you'll see this. The real solution is to parse the query in Koha into a tree, then turn that tree into a proper Elasticsearch query. For obvious reasons (it's hard!), I'm not doing that yet. The other part of the real solution is to catch and parse that ES response and say something sensible, rather than just output it verbatim. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #72 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Katrin Fischer from comment #71)

We currently do it alphabetically - which is kind of ok, but as we pick the facets quite randomly out of 20 records...

Oh, this should be presenting the facets with the most results across the full set of search results.

I understand about not changing behaviour significantly for now, but I think there are a lot of weirdnesses that we shouldn't translate to Elastic Search if we can help it. And our facets are quite weird right now.

Well, they're probably a bit more sane in ES anyway.

Sorry, I don't understand, what do you mean by "(roughly, but I think mostly accurate for the scale of results in a Koha system.)"?

Oh, just that when you get to many results over a clustered system, after a point it starts estimating the counts. Like google's "Results 1-10 of about 1,200,000", it's an estimate, not an exact count. But for smaller result counts it's going to be pretty exact.

The demo database seems to be quite small and from here it's not very fast, so wondering about how performance looks for you. Have you done testing with a bigger database by chance? (100.000+?). We are running into a lot of never-noticed problems right now with the bigger databases and searching.

http://elasticsearch.koha.catalystdemo.net.nz/files/es-search/ Most of the time is spent in Koha::Database. This is something that has to be fixed in general for Koha to use it (alternately, using a persistent environment like Plack where the init only has to happen once.) There is a bit of extra time spent in Catmandu that could be reduced if necessary, we're using it as a fairly thin wrapper over search (as opposed to indexing where it's critical) and it'd be OK to use the elasticsearch libraries directly. However that's a bridge to burn when we get to it. The database I'm using has 8,679 biblios and 14,841 items. So not very large at all. At some stage I might have a look at putting a bigger one in, but that does impact my reindexing time :)

About the number - I think limiting to 10 could be problematic for things like itemtypes, locations and branches. It could be good to have that configurable at some point or an option to just show 'all' for some of the facets.

Sure, but is that a search thing or a Koha thing? I mean, putting hooks in to make this possible is certainly a search thing (they don't exist yet), but the decision on how it should be done is out of scope. I have enough problems to worry about as it is :) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Robin Sheat <robin@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks| |14567 Referenced Bugs: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14567 [Bug 14567] Browse interface for OPAC -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Heather Braum <hbraum@nekls.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hbraum@nekls.org -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #76 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 42031 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42031&action=edit opac_search_for_d_sort_by_title -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #78 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 42033 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42033&action=edit limit_by_book_sort_by_pubdate -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #80 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Jonathan Druart from comment #79)

The first problem I got was to find a MARC21 DB (since the UNIMARC mappings are not defined, I cannot test with an UNIMARC DB).

The UNIMARC mappings should be defined, though not tested.

I have used the one created for the sandboxes (http://git.koha-community.org/gitweb/?p=contrib/global.git;a=blob;f=sandbox/ sql/sandbox1.sql.gz;h=19268bccb43b2a33d5644b7d86cbb1abb323016b;hb=HEAD). But there are only 436 biblios, it's not enough to test some stuffs (facets for instance). Or maybe you can share your DB?

I could, but I think we'll get more useful results from different databases.

Here some notes:

1/ Add deps to C4/Installer/PerlDependencies.pm

Yeah, I'm mostly waiting for things to settle (which they have now.)

2/ The number of tests provided is very low.

Yes, I've been meaning to go back and add a pile more.

3/ catalyst/elastic_search is 1004 commits behind origin/master, please rebase

It's just a tedious process, so I keep putting it off :) should do that soon though.

4/ The message "No 'elasticsearch' block is defined in koha-conf.xml" should be raised before starting the indexation process, and not on commiting the first batch.

Added to my TODO.

5/ You really need to tune the default value for the commit :) commit 100: perl misc/search_tools/rebuild_elastic_search.pl -b 77.57s user 0.86s system 91% cpu 1:25.62 total commit 1000: perl misc/search_tools/rebuild_elastic_search.pl -b 24.68s user 0.52s system 79% cpu 31.595 total For Solr, we used 5000. Yes I know, it's configurable.

I just picked a number and haven't gone back to it. I'm also thinking that maybe dropping the committing entirely and just feeding straight into Catmandu and letting it do its own batching, rather than doubling up on it. More experimentation needed really, but definitely increasing the default is a sensible thing to do. FWIW, committing at 5,000: real 2m14.627s user 1m13.272s sys 0m2.228s 100: real 6m6.280s user 4m45.268s sys 0m2.828s That's a fair difference :)

6/ Verbose does not work as expected, it could be fixed with

Oops. TODOed.

7/ perl -e "use Pod::Checker;podchecker('misc/search_tools/rebuild_elastic_search.pl')"; *** WARNING: empty section in previous paragraph at line 36 in file misc/search_tools/rebuild_elastic_search.pl *** ERROR: =over on line 38 without closing =back at line EOF in file misc/search_tools/rebuild_elastic_search.pl

TODOed.

8/ 2 occurrences of "Solr" reintroduced in installer/data/mysql/sysprefs.sql and koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref

Must have come about when merging. TODOed.

9/ Test! I have launched some searches, with the same DB (the one from the sandbox). On a local using your remote branch and another one using master (sandbox7 provided by BibLibre).

a. Search for 'd' (screentshot opac_search_for_d_sort_by_relevance.png ES on the left, Zebra on the right). Main differences: - 183 vs 182 results (?)

I wouldn't necessarily expect them to be the same, especially for a fairly meaningless search.

- the order is not the same (make sense) - Locations and Places facets are missing

Yeah, they're not faceted yet. Added that to my TODO list before I forget again.

- 6 entries are displayed in the facets for ES (current behavior is 5).

b. Search for 'd', sort by title AZ (screenshot opac_search_for_d_sort_by_title.png) - Zebra displayes only 1 facet

That's probably zebra being wrong then :)

- The order is still completely different

I'm not sure which is right in this case, though I'm doing some work on the sorting at the moment that would allow you to pick which of the fields that end up in title you want to sort by. For example, it might be that ES is putting the ones with a lower series title near the start, even though it displays a different title. That'll be tuneable when I'm done with the current stuff.

c. Search for 'harry', sort by title AZ (screenshot opac_search_for_harry_sort_by_title.png) - 'Show more' links is displayed even if only 2 entries for a facet are available

Thought I'd fixed that, I'll have to have a look again.

- The order is still different ("The discovery of heaven" should be sorted either before Dollhouse (if the is a stopword) either after "Hareios*"

Dollhouse probably has another title field that's actually being used, as noted above.

- The availability is wrong for ES (The item for Dollhouse is not for loan)

Why is it not for loan? Is it by policy, because there are no items, or because all items are issued?

d. Search for Books (limit by item type in the adv search), sort by pubdate (screenshot limit_by_book_sort_by_pubdate.png) - "Return to the last advanced search" link is not displayed

I wonder how it knows to show that... I can't actually find that string in my checkout at all.

- The item types facet contains several entries, which does not make sense

Curious. Are there situations where you have a biblio-level itemtype that differs from the item-level item type, or where one biblio might have multiple items with different item types? At the moment, I think they're all being thrown into one facet pot.

- The number of results highly differ (395 vs 364)

Probably due to biblio-vs-item itemtype selection not being supported yet. If you can find it giving you a record that plain shouldn't match though, that'd be interesting.

- The order is still completely different. I had a look in the index and found: "Pictura murală*" has "pubdate":"||||" (/_search?q=_id:39&pretty) The Korean Go Association's learn to play go "pubdate":"uuuu" (/_search?q=_id:155&pretty) Where do come from these values? Shouldn't be a date, or at least an integer?

Could be the mapping is funny/broken for that. My test system has things like: "pubdate":"1998" though, which implies that it's correct. The actual mapping comes from: INSERT INTO `elasticsearch_mapping` (`indexname`, `mapping`, `facet`, `suggestible`, `type`, `marc21`, `unimarc`, `normarc`) VALUES ('biblios','pubdate',FALSE,FALSE,'','008_/7-10','100a_/9-12','008_/7-10'); On the other hand, it does have: "date-entered-on-file":"61006" which doesn't look right no matter how you carve it.

It's not easy to know what is indexed where. Did you have a look at the indexes configuration page the Solr stuff had? It provided an interface to configure the different mappings, it was very useful.

I haven't yet got to the point where I have the time to make an interface. At the moment it's all configured in elasticsearch_mapping.sql, which is somewhat human readable/editable. After loading the data into a table, it rewrites all those tables into a form that'll be more conducive for having a GUI on top of, but is less human readable. BTW, if you add <trace_to>Stderr</trace_to> to the <elasticsearch> block, it'll dump all the chatter with ES out to stderr, which is useful for seeing what exactly is going on. I warn you, there is a lot there though. Thanks for testing, even if I have a pile more things to fix now :) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #82 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 42062 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42062&action=edit Bug 12478: Fix the UNIMARC and NORMARC indexing 1/ This patch fixes a major issue on UNIMARC and NORMARC installations (Compulsive c/p on insert into insert into search_marc_to_field for unimarc and normarc) 2/ After fixing that, I got invalid marc path at /usr/share/perl5/Catmandu/Fix/marc_map.pm line 35. on reindexing. It was caused by some "" values instead of NULL. 3/ Then, a "duplicate entry" error was caused by 3 lines for Local-number. It seems to be an error, 1 is enough (I suppose 0909 was an error too). Note that the following is not fixed by this patch: MariaDB [koha_es_unimarc]> insert into search_field (name, type) select distinct mapping, type from elasticsearch_mapping; Query OK, 73 rows affected, 57 warnings (0.05 sec) Records: 73 Duplicates: 0 Warnings: 57 MariaDB [koha_es_unimarc]> show warnings; +---------+------+--------------------------------------------+ | Level | Code | Message | +---------+------+--------------------------------------------+ | Warning | 1265 | Data truncated for column 'type' at row 1 | -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #84 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 42064 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42064&action=edit Bug 12478: Fix the verbose flag on reindexing -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #86 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 42066 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42066&action=edit Bug 12478: Remove Solr occurrences reintroduced by a previous patch -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #88 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 42068 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42068&action=edit Bug 12478: Do not display the 'Show more' links if no more facet available -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #90 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Something else, there is a sort issue in the facets: [Some entries] Zeitoun, Ariel, Ó Cadhain, Máirtín. Ślez, Ts.. Ó should be after O, not after Z. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #92 from Peter Zhao <peterzhaonj@163.com> --- I'v install a ES Koha. Index works well and ES server can search records, but the Opac cannot search. "No results found! You did not specify any search criteria. Error: Unable to perform your search. Please try again. " -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #94 from Peter Zhao <peterzhaonj@163.com> --- (In reply to Robin Sheat from comment #93)

(In reply to Peter Zhao from comment #92)

...

I'v install a ES Koha. Index works well and ES server can search records, but the Opac cannot search.

Can you provide more detail? Comment #80, at the bottom, shows how to debug the traces between ES and Koha. This will tell you what search request is actually being made.

koha-opac-error_log shows: [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] # Request to: http://localhost:9200, referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] curl -XHEAD 'http://localhost:9200/koha_biblios?pretty=1', referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] , referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] # Response: 200, Took: 14 ms, referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] # 1, referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 2015] opac-search.pl: [Serializer] ** encountered object '1', but neither allow_blessed nor convert_blessed settings are enabled at /usr/local/share/perl/5.14.2/Search/Elasticsearch/Role/Serializer/JSON.pm line 24., referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 2015] opac-search.pl: , called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/local/share/perl/5.14.2/Catmandu/Store/ElasticSearch/Bag.pm line 127. With vars: {'var' => {'from' => 0,'query' => {'query_string' => {'fuzziness' => 'auto','default_field' => '_all','query' => '(title:best)','default_operator' => 'AND','lenient' => bless( do{\\(my $o = 1)}, 'JSON::PP::Boolean' )}},'size' => 20,'facets' => {'subject' => {'terms' => {'field' => 'subject__facet'}},'author' => {'terms' => {'field' => 'author__facet'}},'itype' => {'terms' => {'field' => 'itype__facet'}}}}}, referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 2015] opac-search.pl: Use of uninitialized value $error in concatenation (.) or string at /home/koha/kohaes/opac/opac-search.pl line 578., referer: http://127.0.1.1/ -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #96 from Peter Zhao <peterzhaonj@163.com> --- (In reply to Robin Sheat from comment #95)

(In reply to Peter Zhao from comment #94)

...

[Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 2015] opac-search.pl: [Serializer] ** encountered object '1', but neither allow_blessed nor convert_blessed settings are enabled at /usr/local/share/perl/5.14.2/Search/Elasticsearch/Role/Serializer/JSON.pm line 24., referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 2015] opac-search.pl: , called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/local/share/perl/5.14.2/Catmandu/Store/ElasticSearch/Bag.pm line 127. With vars: {'var' => {'from' => 0,'query' => {'query_string' => {'fuzziness' => 'auto','default_field' => '_all','query' => '(title:best)','default_operator' => 'AND','lenient' => bless( do{\\(my $o = 1)}, 'JSON::PP::Boolean' )}},'size' => 20,'facets' => {'subject' => {'terms' => {'field' => 'subject__facet'}},'author' => {'terms' => {'field' => 'author__facet'}},'itype' => {'terms' => {'field' => 'itype__facet'}}}}}, referer: http://127.0.1.1/

Interesting. What do you get as the output of:

$ perl -MData::Dumper -MJSON -e 'print Dumper JSON::true;'

Also, what about:

perl -MJSON::XS::Boolean -e ''

Thirdly, does installing libjson-xs-perl make things work?

koha@koha:~$ perl -MData::Dumper -MJSON -e 'print Dumper JSON::true;' $VAR1 = bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ); koha@koha:~$ perl -MJSON::XS::Boolean -e '' koha@koha:~$ koha@koha:~$ sudo apt-get install libjson-xs-perl [sudo] password for koha: 正在读取软件包列表... 完成 正在分析软件包的依赖关系树 正在读取状态信息... 完成 libjson-xs-perl 已经是最新的版本了。 libjson-xs-perl 被设置为手动安装。 升级了 0 个软件包，新安装了 0 个软件包，要卸载 0 个软件包，有 0 个软件包未被升级。 (In reply to Robin Sheat from comment #95)

(In reply to Peter Zhao from comment #94)

...

[Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 2015] opac-search.pl: [Serializer] ** encountered object '1', but neither allow_blessed nor convert_blessed settings are enabled at /usr/local/share/perl/5.14.2/Search/Elasticsearch/Role/Serializer/JSON.pm line 24., referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 2015] opac-search.pl: , called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/local/share/perl/5.14.2/Catmandu/Store/ElasticSearch/Bag.pm line 127. With vars: {'var' => {'from' => 0,'query' => {'query_string' => {'fuzziness' => 'auto','default_field' => '_all','query' => '(title:best)','default_operator' => 'AND','lenient' => bless( do{\\(my $o = 1)}, 'JSON::PP::Boolean' )}},'size' => 20,'facets' => {'subject' => {'terms' => {'field' => 'subject__facet'}},'author' => {'terms' => {'field' => 'author__facet'}},'itype' => {'terms' => {'field' => 'itype__facet'}}}}}, referer: http://127.0.1.1/

Interesting. What do you get as the output of:

$ perl -MData::Dumper -MJSON -e 'print Dumper JSON::true;'

Also, what about:

perl -MJSON::XS::Boolean -e ''

Thirdly, does installing libjson-xs-perl make things work?

koha@koha:~$ perl -MData::Dumper -MJSON -e 'print Dumper JSON::true;' $VAR1 = bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ); koha@koha:~$ perl -MJSON::XS::Boolean -e '' koha@koha:~$ koha@koha:~$ sudo cpanm JSON::XS After "JSON::XS" was installed, Opac works well. Thanks a lot! -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #98 from Peter Zhao <peterzhaonj@163.com> --- ES Opac cannot support Chinese language? I try to search Chinese word in http://elasticsearch.koha.catalystdemo.net.nz/ and my ES, they also show "Software error". Software error: Can't escape \x{57FA}, try uri_escape_utf8() instead at /opt/kohaclones/elasticsearch/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 217. For help, please send mail to the webmaster ([no address given]), giving this error message and the time and date of the error. Software error: Can't escape \x{57FA}, try uri_escape_utf8() instead at /home/koha/kohaes/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 217. For help, please send mail to the webmaster (webmaster@koha), giving this error message and the time and date of the error. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #100 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Robin Sheat from comment #91)

(In reply to Jonathan Druart from comment #81)

...

Note the following: MariaDB [koha_es_unimarc]> insert into search_field (name, type) select distinct mapping, type from elasticsearch_mapping; Query OK, 73 rows affected, 57 warnings (0.05 sec) Records: 73 Duplicates: 0 Warnings: 57

MariaDB [koha_es_unimarc]> show warnings; +---------+------+--------------------------------------------+ | Level | Code | Message | +---------+------+--------------------------------------------+ | Warning | 1265 | Data truncated for column 'type' at row 1 |

Hmm, I remember that, but I'm not 100% sure it mattered. Could be wrong though.

It's caused by the fact that you insert a an empty string into an enum field. I am not sure about the consequences.

Here it is:

http://elasticsearch.koha.catalystdemo.net.nz/files/koha_es_marc21.sql.bz2

it's not the best data, but it's good enough for messing about with.

Great, thanks. Another set of data :)

...

It comes from the 008

...

"Pictura murală*" has "pubdate":"||||" (/_search?q=_id:39&pretty) 008 090409|||||||||xx |||||||||||||| ||und|| The Korean Go Association's learn to play go "pubdate":"uuuu" 008 971030muuuu9999nyua 000 0 eng

But the index should not contain an invalid date.

Hmm. I don't know if we can put validation into the fixer rules. I'll have to explore that some further. Possibly also telling ES that this must be a number could cause bad data to get rejected, but it may reject the whole record, not sure.

Do you happen to know how zebra handles that?

Absolutely no idea.

...

For Solr (you can find the code on the BibLibre repo at https://git.biblibre.com/biblibre/koha_biblibre/commits/dev/solr Browse C4/Search/), we used a system of plugins. And there is a Date plugin (https://git.biblibre.com/biblibre/koha_biblibre/blob/ bd38ce1811289fcfbd75a37ec99fc4cd3c5d37f4/C4/Search/Plugins/Date.pm) which does this job. A plugin can be linked to a mapping.

We probably can't directly reuse that, at present we're using Catmandu do do the data conversion and interfacing with ES for the most part. But it's possible I can hook something in somewhere.

We will have to do some data pre-processing before indexing the records. I need to learn more about ES, but with Solr we had to process the date values for date type mappings. Otherwise it is not possible to correctly request on this index (for instance dates range, or sort by, etc.). By the way, the date type is only used on acqdate and copydate, why not on other dates (at least pubdate)?

(In reply to Jonathan Druart from comment #90)

...

Something else, there is a sort issue in the facets:

[Some entries] Zeitoun, Ariel, Ó Cadhain, Máirtín. Ślez, Ts..

Ó should be after O, not after Z.

Line 573 of opac/opac-search.pl does a sort with cmp, which isn't very unicode aware. I'm putting that in the not-my-problem bin as it's in upstream :)

Yes, and, IMO, there is a design issue here. We should not reuse the pl and tt files. How do you plan to add features that Zebra cannot provide? :) Not sure it will be maintainable to add conditions (if SE == 'ES') in the TT. For instance, for the facets, we would like to display them as ES retrieve them (order by most used), and add the number of occurrences. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #102 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Ok, just saw the comment about the date format: Koha/ElasticSearch.pm 185 # TODO be aware of date formats, but this requires pre-parsing 186 # as ES will simply reject anything with an invalid date. We would pre-process the data at this point, if a "plugin is linked to" (to define) this field. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #104 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Peter Zhao from comment #103)

After I changed "uri_escape" to "uri_escape_utf8", ES can index and Opac can search Chinese in MARC21 structure.

But it does not work in UNIMARC structure. It shows messy code.

The code paths between MARC21 and UNIMARC shouldn't differ in ways that would make a difference here. The only place where it comes up is determining that XXXy goes into "author" or whatever. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #106 from Robin Sheat <robin@catalyst.net.nz> --- Just a headsup that I'll be out of touch until about the 17th of September, so that's why there'll be radio silence for a while :) Jonathan, I was hoping to pull your patches in on top of my sorting fixes, unfortunately I didn't quite get them finished in time. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #108 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 42393 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=42393&action=edit Bug 12478: Fix encoding issues on indexing There were encoding issues on indexing, the way to build a marc record was the old way (retrieved from the noxml style from rebuild_zebra). Here we should either c/p the code from C4::Biblio::GetMarcBiblio or call it directly (what this patch does). -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #110 from Peter Zhao <peterzhaonj@163.com> --- (In reply to Jonathan Druart from comment #109)

Peter, the last patch should fix your issue.

Jonathan, it works well. Thanks a lot. ES indexing and Opac works wonderful. But in staff interface when I input non-latin words (e.g. Chinese) to search (search the catalog), it shows "Software error:Cannot decode string with wide characters at /usr/lib/i386-linux-gnu/perl/5.20/Encode.pm line 215. For help, please send mail to the webmaster (webmaster@koha), giving this error message and the time and date of the error. " If I input English words, it can show results, but when I click the record, it shows "The record you requested does not exist ()." -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Bob Birchall <bob@calyx.net.au> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bob@calyx.net.au -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #113 from Robin Sheat <robin@catalyst.net.nz> --- Jonathan, I've added all your patches to the branch. Thanks! -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #114 from Mirko Tietgen <mirko@abunchofthings.net> --- (In reply to Robin Sheat from comment #80)

...

3/ catalyst/elastic_search is 1004 commits behind origin/master, please rebase

It's just a tedious process, so I keep putting it off :) should do that soon though.

Now? Please. :) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #116 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Robin Sheat from comment #115)

I have a rebase done, though it's untested.

Now it's had a quick test and some things have been fixed. It appears on the surface to be working again. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #117 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- I am getting error on the DB update process: C4::Installer::load_sql returned the following errors while attempting to load /home/koha/src/installer/data/mysql/elasticsearch_mapping.sql: ERROR 1062 (23000) at line 294: Duplicate entry 'biblios-marc21-490a' for key 'index_name_2' ERROR 1062 (23000) at line 294: Duplicate entry 'biblios-marc21-490a' for key 'index_name_2' -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #119 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Does the authorities index is supposed to work? I get Use of uninitialized value $id in concatenation (.) or string at misc/search_tools/rebuild_elastic_search.pl line 173. after a perl misc/search_tools/rebuild_elastic_search.pl -a -v and on /koha-es_authorities/_search?q=*&pretty "_index" : "koha-es_authorities", "_type" : "data", "_id" : "7c8462c8-c813-4013-9ab7-1ca77069bbc4", -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #121 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Jonathan Druart from comment #119)

Does the authorities index is supposed to work?

I get Use of uninitialized value $id in concatenation (.) or string at misc/search_tools/rebuild_elastic_search.pl line 173.

That shouldn't happen, I think there was something underlying that changed in the rebase. Lemme sort that out... -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #123 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Robin, just retried now with and everything works fine. Not sure where did it come from, but it looks to be fixed! Just got: Upgrade to XXX done (Bug 12478 - set up elasticsearch tables) DBD::mysql::db do failed: Can't DROP 'index_name_2'; check that column/key exists at installer/data/mysql/updatedatabase.pl line 10967.DBD::mysql::db do failed: Duplicate column name 'label' at installer/data/mysql/updatedatabase.pl line 10970. Upgrade to XXX done (Bug 12478/14899 - DB Changes to the elasticsearch tables) when updating the DB. You have updated the structure in the elasticsearch_mapping.sql file, so no need to remove the unique key and add the label field in the last updatedatabase entry. Not a big deal anyway :) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #125 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Robin, Another set of patches is coming. 1/ Reintroduce the SearchEngine pref 2/ Fix to correct the number of facets displayed (again) 3/ Remove an unnecessary "&limit=" (minor) 4/ Replace the Koha::ItemType[s] classes you have introduced with a Koha::Object[s] way 5/ Display description instead of code for locations in facets. About 4: I know you are now agreeing completely with Koha::Object[s] and how it has been implemented. But I think we need to have an homogeneous code, and all new classes in the Koha namespace should follow the same directions. I would please you to have a look at bug 14828 (and the see also ones), which introduces the Koha::ItemType[s] classes as well and rewrites the admin/itemtypes.pl script, to see how easy it is to use it. To avoid future conflicts and don't play to a race against each other we should agree on this kind of decisions now. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #127 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 43128 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=43128&action=edit Bug 12478: (follow-up) Display the correct number of facets This patch fixes a side-effect of the previous "Display the correct number of facets (5 instead of 6)" patch. If you ask for more facets ('show more' link) and that the number of facets is > 5 but < 10, 1 facet will be missing. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #129 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 43130 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=43130&action=edit Bug 12478: Remove Koha::ItemType[s] Class::Accessor classes -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #131 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 43132 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=43132&action=edit Bug 12478: Use the new Koha::ItemTypes to retrieve itypes descriptions -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #133 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 43134 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=43134&action=edit Bug 12478: facets - Display description instead of code for locations -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #135 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> ---

(In reply to Jonathan Druart from comment #90)

...

...

Something else, there is a sort issue in the facets:

[Some entries] Zeitoun, Ariel, Ó Cadhain, Máirtín. Ślez, Ts..

Ó should be after O, not after Z.

Line 573 of opac/opac-search.pl does a sort with cmp, which isn't very unicode aware. I'm putting that in the not-my-problem bin as it's in upstream :)

Yes, and, IMO, there is a design issue here. We should not reuse the pl and tt files. How do you plan to add features that Zebra cannot provide? :) Not sure it will be maintainable to add conditions (if SE == 'ES') in the TT. For instance, for the facets, we would like to display them as ES retrieve them (order by most used), and add the number of occurrences.

In the last patch I suggest to display the facet terms in the same order as ES build them (i.e. most used first). -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #43141|0 |1 is obsolete| | --- Comment #137 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 43142 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=43142&action=edit Bug 12478: Remove empty limit parameter If no limit are passed, the url will contain '&limit=' anyway. It is not necessary and can be avoided easily Test plan: 1/ Search for a term in your catalogue 2/ Hover over a link in the facet area 3/ The link is cgi-bin/koha/opac-search.pl?idx=kw&q=your_term&limit=&limit=[...] With this patch, the empty limit parameter does not appear. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #139 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Jonathan Druart from comment #123)

You have updated the structure in the elasticsearch_mapping.sql file, so no need to remove the unique key and add the label field in the last updatedatabase entry. Not a big deal anyway :)

Oh, yes. That should have been removed. (In reply to Jonathan Druart from comment #125)

I know you are now agreeing completely with Koha::Object[s] and how it has been implemented. But I think we need to have an homogeneous code, and all new classes in the Koha namespace should follow the same directions. I would please you to have a look at bug 14828 (and the see also ones), which introduces the Koha::ItemType[s] classes as well and rewrites the admin/itemtypes.pl script, to see how easy it is to use it. To avoid future conflicts and don't play to a race against each other we should agree on this kind of decisions now.

Sure, that's fine. Consistency is generally better than reinventing everything. (In reply to Jonathan Druart from comment #135)

In the last patch I suggest to display the facet terms in the same order as ES build them (i.e. most used first).

While I've added this, in general I'm avoiding changing of behaviour too much. We can always remove it later if it's decided we don't want it (and personally, I think it'll be better.) All these patches are added. Thanks Jonathan! FYI, I'm looking into the notforloan stuff at the moment. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #141 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Robin, Thanks for your responsiveness! Yesterday I have read a bit of doc about facets and on the Terms Facet page of the ES doc ( https://www.elastic.co/guide/en/elasticsearch/reference/current/search-facet...), there is a big warning on top " Warning Facets are deprecated and will be removed in a future release. You are encouraged to migrate to aggregations instead.". Did you have time to look into the terms aggregations? Or maybe is it what you are already using? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #143 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Robin, Maybe I have not drunk enough tea this morning but... I am trying to improve the mappings area to have a consistent interface to manage them. The idea is to 1) move the elastic_mapping.sql to a elastic_mapping.json file (easier to modify and read), 2) provide methods to serialize/unserialize mappings and then 3) introduce a backup/import/reset mappings feature and finally 4) make easier the mapping progression to get a good basis to use ES. I have managed to create a json file from the sql file, the structure is something like: { biblio => { title => { label => 'Title', type => 'string', mappings => [ { suggestible => 1, facet => 1, marc21 => '245a', unimarc => '200a', normarc => '245a', }, ] }, } And I have some questions :) - Do you agree with the idea? - Don't you think the index_name should be a column of the search_fields table? - Some of the fields don't have a type, should we assign "string" as the default value? - wordings: 'sortable' and 'facetable' sounds more appropriate than 'sort' and 'facet' - (/me is clearing his throat) I think that all the mappings of a field should be removed if the field is removed. In other word, there is a 1-n relationship beetwen search_field and search_marc_map, which means that the join table (search_marc_to_field) is not needed and we could simplify the structure removing it. I am going to wait for an answer before starting anything :) -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #145 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Jonathan Druart from comment #141)

Did you have time to look into the terms aggregations? Or maybe is it what you are already using?

Yeah, I'm aware of that. Unfortunately, it only became deprecated after I'd implemented it, and I haven't got around to going back and reworking it. I don't think it'll be a big change, it's possible it's sufficient to just switch the ES type we're using. Anyway, I've put it on the "deal with later" pile. (In reply to Jonathan Druart from comment #142)

On a unimarc installation, there are no mapping at all for authorities. If an unimarc user follows this bug report...

Working out the MARC21 mappings was tedious enough :) (In reply to Jonathan Druart from comment #143)

Maybe I have not drunk enough tea this morning but... I am trying to improve the mappings area to have a consistent interface to manage them. The idea is to 1) move the elastic_mapping.sql to a elastic_mapping.json file (easier to modify and read), 2) provide methods to serialize/unserialize mappings and then 3) introduce a backup/import/reset mappings feature and finally 4) make easier the mapping progression to get a good basis to use ES.

I have managed to create a json file from the sql file, the structure is something like:

{ biblio => { title => { label => 'Title', type => 'string', mappings => [ { suggestible => 1, facet => 1, marc21 => '245a', unimarc => '200a', normarc => '245a', }, ] }, }

And I have some questions :) - Do you agree with the idea?

Well... I don't know. Though I'm not a fan of that structure really, as it's not ideal, and is a bit more limited. Also, in this case you can't have more than one title, but that's not really the big issue. Mostly it's just a very denormalised view of the data. Better for manually editing an SQL file, but not really so good for a computer to use. This is why the SQL file has the data in that form and then normalises it in the database.

- Don't you think the index_name should be a column of the search_fields table?

Yes, it should be kept with search_field.name as it's effectively more information needed to describe where something gets stored.

- Some of the fields don't have a type, should we assign "string" as the default value?

I'd like to not just because that implies that they've consciously been made strings. Ideally as time goes on, people will decide that this is a date, and this is a ... IP address or something, and add those as types while putting the logic in to handle it. So, if a type is unspecified, then it gets treated like a string by default, but it really means "we haven't decided yet."

- wordings: 'sortable' and 'facetable' sounds more appropriate than 'sort' and 'facet'

hmm. I don't really mind either way. My thinking was that "facet" and "sort" were easier to type. But I broke the consistency because "suggest" seemed weird. I don't object to any of them changing.

- (/me is clearing his throat) I think that all the mappings of a field should be removed if the field is removed. In other word, there is a 1-n relationship beetwen search_field and search_marc_map, which means that the join table (search_marc_to_field) is not needed and we could simplify the structure removing it.

I had a good reason for doing many-to-many. Let me see if I can remember it... Oh wait, I documented it: -- This joins the two search tables together. We can have any combination: -- one marc field could have many search fields (maybe you want one value -- to go to 'author' and 'corporate-author) and many marc fields could go -- to one search field (e.g. all the various author fields going into -- 'author'.) If you remove the many-to-many relationship then you end up with duplication/denormalisation. My thinking behind the UI is that you might have, say, a list of all the fields and under them, a set of all the MARC fields that map to it. Or perhaps the inverse. I hadn't really thought about it too much, but a properly denormalised relational structure means that we have the maximum amount of flexibility. The only improvement to the structure in this respect is that the sort, facet, suggest things should really be at the join level. I have a feeling I considered that, then decided it risked crossing the line into too fiddly, but it would get more power out of it. At the moment you'd have to duplicate the MARC details if you want different values for those three which isn't ideal. (In reply to Jonathan Druart from comment #144)

Ha, something else: the biblionumber should be a field of the biblios index.

Oh, it's embedded as the ID on the ES record. There's no point duplicating it as its own field, but it's reasonable to copy it out as a post-process step and put it into a biblionumber field. We don't have a reliable -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #147 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Robin Sheat from comment #145)

(In reply to Jonathan Druart from comment #143)

...

Maybe I have not drunk enough tea this morning but... I am trying to improve the mappings area to have a consistent interface to manage them. The idea is to 1) move the elastic_mapping.sql to a elastic_mapping.json file (easier to modify and read), 2) provide methods to serialize/unserialize mappings and then 3) introduce a backup/import/reset mappings feature and finally 4) make easier the mapping progression to get a good basis to use ES.

I have managed to create a json file from the sql file, the structure is something like:

{ biblio => { title => { label => 'Title', type => 'string', mappings => [ { suggestible => 1, facet => 1, marc21 => '245a', unimarc => '200a', normarc => '245a', }, ] }, }

And I have some questions :) - Do you agree with the idea?

Well... I don't know. Though I'm not a fan of that structure really, as it's not ideal, and is a bit more limited. Also, in this case you can't have more than one title, but that's not really the big issue. Mostly it's just a very denormalised view of the data. Better for manually editing an SQL file, but not really so good for a computer to use. This is why the SQL file has the data in that form and then normalises it in the database.

I don't understand the problem with the structure, you could have several mappings (it's an arrayref of hashrefs). With this structure I could insert exactly the same data in the tables (except if I missed something...).

...

- Some of the fields don't have a type, should we assign "string" as the default value?

I'd like to not just because that implies that they've consciously been made strings. Ideally as time goes on, people will decide that this is a date, and this is a ... IP address or something, and add those as types while putting the logic in to handle it. So, if a type is unspecified, then it gets treated like a string by default, but it really means "we haven't decided yet."

So todo later :)

...

- wordings: 'sortable' and 'facetable' sounds more appropriate than 'sort' and 'facet'

hmm. I don't really mind either way. My thinking was that "facet" and "sort" were easier to type. But I broke the consistency because "suggest" seemed weird. I don't object to any of them changing.

Not a big deal but better sooner than later.

...

- (/me is clearing his throat) I think that all the mappings of a field should be removed if the field is removed. In other word, there is a 1-n relationship beetwen search_field and search_marc_map, which means that the join table (search_marc_to_field) is not needed and we could simplify the structure removing it.

I had a good reason for doing many-to-many. Let me see if I can remember it...

Oh wait, I documented it:

-- This joins the two search tables together. We can have any combination: -- one marc field could have many search fields (maybe you want one value -- to go to 'author' and 'corporate-author) and many marc fields could go -- to one search field (e.g. all the various author fields going into -- 'author'.)

If you remove the many-to-many relationship then you end up with duplication/denormalisation. My thinking behind the UI is that you might have, say, a list of all the fields and under them, a set of all the MARC fields that map to it. Or perhaps the inverse. I hadn't really thought about it too much, but a properly denormalised relational structure means that we have the maximum amount of flexibility. The only improvement to the structure in this respect is that the sort, facet, suggest things should really be at the join level. I have a feeling I considered that, then decided it risked crossing the line into too fiddly, but it would get more power out of it. At the moment you'd have to duplicate the MARC details if you want different values for those three which isn't ideal.

Yes, it's related to the index_name unique key discussion we had last week. We should either move sort, facet, suggest to the join table or remove it, but not keep the current structure. I am not sure about the gain of having the three tables, we could still know what fields are mapped with this MARC field or the inverse (same marc_field values). Anyway, the current structure force us to duplicate the MARC details, because of the sort, facet, suggest, which could differ.

(In reply to Jonathan Druart from comment #144)

...

Ha, something else: the biblionumber should be a field of the biblios index.

Oh, it's embedded as the ID on the ES record. There's no point duplicating it as its own field, but it's reasonable to copy it out as a post-process step and put it into a biblionumber field. We don't have a reliable

Indeed we could add it later, but don't you think it's worth to let the librarians (and devs) search something like "biblionumber:42", which is a more familiar term than "ID"? -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #149 from Robin Sheat <robin@catalyst.net.nz> --- (In reply to Robin Sheat from comment #91)

...

The item is a "Visual Materials" which has a itemtype.notforloan flag set.

Good to know, I've not tested that case yet.

Well, I spent far too long staring at that, only to find it's been working all along. You can set it on the itemtype and it should have always worked (as that's done in post-processing, which is the same between zebra and es), and all it needed to work at the item level was a mapping from the notforloan field into es to be added. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #151 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Found something else: In the sql file you have Heading-main vs Heading-Main. record-source vs Record-Source. For the first one, it's a typo (both for authorities). For the second one (it's a typo too, but), and for Local-number, you have the search_field.name which will be mapped with biblios and authorities fields. Does it make sense in your mind? In mine it's a bit weird (but maybe I need a pint). -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #153 from Robin Sheat <robin@catalyst.net.nz> --- I've created a zebra-running version to help shake out the bugs that have accumulated in there. It's at: http://zebra.koha.catalystdemo.net.nz/ and is exactly the same code as the elasticsearch version, but with the SearchEngine preference set to 'Zebra'. It isn't working right now, I've probably broken something. That's what I'll be fixing for the next little while. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #155 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 43346 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=43346&action=edit Bug 12478: Add the yaml mappings file This should facilitate the modification of the mappings. The yaml format is more readable than sql. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #157 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 43348 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=43348&action=edit Bug 12478: Move mapping attributes to the join table To make things consistent, the attributes of the mappings (sort, suggestible, facet) should be moved to the join tables (search_marc_to_field). Indeed the search_marc_map could represent a mapping for several search fields with different attributes. Before this change it was not possible. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #159 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 43350 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=43350&action=edit Bug 12478: Manually add the many_to_many relationships This will create the add_to_* accessors for us. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #33137|0 |1 is obsolete| | Attachment #33138|0 |1 is obsolete| | Attachment #33139|0 |1 is obsolete| | Attachment #33140|0 |1 is obsolete| | Attachment #33141|0 |1 is obsolete| | Attachment #42062|0 |1 is obsolete| | Attachment #42063|0 |1 is obsolete| | Attachment #42064|0 |1 is obsolete| | Attachment #42065|0 |1 is obsolete| | Attachment #42066|0 |1 is obsolete| | Attachment #42067|0 |1 is obsolete| | Attachment #42068|0 |1 is obsolete| | Attachment #42069|0 |1 is obsolete| | Attachment #42392|0 |1 is obsolete| | Attachment #42393|0 |1 is obsolete| | Attachment #43127|0 |1 is obsolete| | Attachment #43128|0 |1 is obsolete| | Attachment #43130|0 |1 is obsolete| | Attachment #43131|0 |1 is obsolete| | Attachment #43132|0 |1 is obsolete| | Attachment #43133|0 |1 is obsolete| | Attachment #43134|0 |1 is obsolete| | Attachment #43140|0 |1 is obsolete| | Attachment #43142|0 |1 is obsolete| | Attachment #43148|0 |1 is obsolete| | --- Comment #161 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 43352 --> http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=43352&action=edit Bug 14899: Add a link to the new page in the admin -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #162 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Hi Robin, I think I did it! :) Last patch set move the mapping attributes to the join table and provide a yaml file to manage the default mappings. I will test it a bit more tomorrow, to be sure I have not introduced regression. Let me know if it goes in the direction of what you had in mind! I am also attaching patches on bug 14899, the interface is now much more consistent. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #164 from Robin Sheat <robin@catalyst.net.nz> --- Jonathan, you're not putting correct copyright headers into the files you're creating/modifying. You need to. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #166 from Robin Sheat <robin@catalyst.net.nz> --- Commit cd4905c2 (Display facet terms ordered by number of occurrences) introduces a bug when searching with zebra (i.e. it makes it break.) I will roll it back until we get a better solution. Feel free to unroll it back if you get the chance to make one. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #168 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- (In reply to Robin Sheat from comment #166)

Commit cd4905c2 (Display facet terms ordered by number of occurrences) introduces a bug when searching with zebra (i.e. it makes it break.) I will roll it back until we get a better solution. Feel free to unroll it back if you get the chance to make one.

Oops, sorry about that! The last patch should work now. -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 kohayu <long_sam.tw@yahoo.com.tw> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |long_sam.tw@yahoo.com.tw -- You are receiving this mail because: You are watching all bug changes.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12478 --- Comment #171 from kohayu <long_sam.tw@yahoo.com.tw> --- Dear folk, I git clone git://git.catalyst.net.nz/koha.git . os - debian 7.9 wheezy , x86_64 Linux debian 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u4 x86_64 GNU/Linux koha version - 3.21.00.030 perl - 5.014002 mysql - mysql Ver 14.14 Distrib 5.5.44, for debian-linux-gnu (x86_64) using readline 6.2 Elasticsearch version: 1.7.2 JVM name: Java HotSpot(TM) 64-Bit Server VM JVM vendor: Oracle Corporation JVM version: 25.60-b23 Java version: 1.8.0_60 I use bulkmarcimport.pl to import marc data, But import 1,000 records every time , it occurred "................[NoNodes] ** No nodes are available: [http://127.0.0.1:9200], called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/local/share/perl/5.14.2/Catmandu/Store/ElasticSearch.pm line 61.root@debian:/usr/share/koha/bin/migration_tools#" This job stopped. Has anyone see the same condition? BR. longshan -- You are receiving this mail because: You are watching all bug changes.