Solr / zebra / search in Koha 3.10 => starting a workgroup
Hello all, As you know, our main goal for the oct12 release of Koha is to introduce solr as an alternate search engine. BibLibre already explained which improvements will be added by this search engine on the blog page: http://drupal.biblibre.com/en/blog/entry/solr-developments-for-koha During the hackfest in Marseille, a group of 4 persons (Claire, Henri-Damien, Juan and Zeno) worked on how this work should be done to be introduced smoothly. The first goal being that a library wanting to run zebra still could. As some librarians could want to use another search engine than zebra or solr, we want to follow a path that would result in a better modularity. I also think that most of us agree that current search code is ugly & very hard to maintain/improve. The hackfesters have produced a drawing explaining how we could name the different packages: https://docs.google.com/a/biblibre.com/drawings/d/1ZdsQsoThYgIVSgH3LqgRZy17x..., with a page on the wiki: http://wiki.koha-community.org/wiki/Switch_to_Solr_RFC#.23kohahack12 In this drawing (read from bottom to top), there are 2 main layers "Search" and "Index", that are reponsible of doing searches and doing indexing. The "Conf" object will be responsible to retrieve the configuration (current getIndexes), the "Query" object would be responsible to build the query in SearchEngine grammar, the "Plugin" object would be reponsible to deal with records before indexing (like normalizing data) Claire (from BibLibre) made a first implementation of this organization on github: https://github.com/clrh/wip-searchengine-layer/tree/master/lib/SearchEngine. Juan (from xercode), also worked on this organization, on the zebra side. His code is available also on github: https://github.com/xercode/Data-SearchEngine-Zebra. Now, Henri-Damien is continuing the work for implementing zebra with this global structure. In the meantime, 2 other directions have been followed: * Frédéric (Demians, from Tamil) wrote a daemon for zebra indexing (see http://git.tamil.fr/?p=Koha-Contrib-Tamil;a=summary), that resulted in bug http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7759, that document how to introduce this daemon for indexing. Liz (and maybe others) are using it without any problem. This git repository introduces some other tools, but what they effectively do is not completely clear to me (Frédéric, if you want, to add some info...) * Galen (Charlton, from Equinox) wrote some code that you can see in http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7818 and is in "needs signoff" status. The description of the bug includes a lot of things: DOM indexing for biblios (and a tool to automatically write the DOM xsl from the record.abs), and a normalizer for datas, an indexer (Koha::Indexer). Unless I've missed something (Galen, tell if I'm wrong): for now, only the DOM indexing is submitted, normalizer and indexer are not. What we all agree about: we should have a clearer way to: Normalize / Index / Search in Koha. That's great ! The structure described by the hackfester is great because it's independent from the SearchEngine you use. I think large portions (if not all) of Koha::Contrib::Tamil could be used to write the zebra indexing layer. I also think that The DOM indexing part of what Galen has submitted can be signed-off & pushed without any risk, but the normalize and indexer parts will need coordination to avoid having BibLibre/xercode working in a direction, and Galen working in another. I really like the idea of having normalizer not necessary being MARC; that could be useful in the future. That's why I propose to organize an IRC meeting (date and time to define, but that will be in Europe afternoon / US morning) with all volunteers to coordinate their efforts. I think this meeting should be regular (monthly ?) After each meeting, a summary of the conclusions would be made on the wiki and posted on this mailing-list. My proposition: if you're interested by participating to this effort, please answer to this mail. (I'll then start a doodle to find a proper time. I propose 2 hours for the duration of the 1st meeting, then, hopefully, shorter meetings) -Juan/Galen/Zeno, you're considered as being interested by this topic ;-) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
The hackfesters have produced a drawing explaining how we could name the different packages:
The classes hierarchy seems to rely on Data::SearchEngine module as abstraction layer: https://metacpan.org/release/Data-SearchEngine Are you it touch with the module author? He could give us interesting feedback on his module. What kind of implementation has he done? Is there any other implementation done by someone else than his author? Is it generic enough to be used in Koha context? There is a ElasticSearch implementation of Data::SearchEngine: https://github.com/gphat/data-searchengine-elasticsearch In implementation notes, we can read: ElasticSearch's query DSL is large and complex. It is not well suited to abstraction by a library like this one. As such you will almost likely find this abstraction lacking. Expect it to improve as the author uses more of ElasticSearch's features in applications.
* Frédéric (Demians, from Tamil) wrote a daemon for zebra indexing (see http://git.tamil.fr/?p=Koha-Contrib-Tamil;a=summary), that resulted in bug http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7759, that document how to introduce this daemon for indexing. Liz (and maybe others) are using it without any problem.
I don't think there is a lot to do to have an abstraction layer for indexing with SolR/Zebra/xx. Koha::Indexer | +-- Koha::Indexer::SolR +-- Koha::Indexer::Zebra A indexer is then able to index partially (queued) or fully biblio or authority records. A command line indexing script is nothing more than a wrapper to this class. It's even possible to run the task from the web interface. There is a 'watcher' associated with the indexer which could communicate asynchronously with a WUI via a JavaScript callback function. As with rebuild_zebra.pl, indexing is a two step process: (1) export record and (2) index records. Since record format/syntax to be sent to the search engine may vary: XML, ISO2709, JSON (ElasticSearch), the exporter must also be a generic class subclassed by a specific class for each search engine, implementing normalization processing. I'd need to see how it works now in SolR/Biblibre branch. For me, the most undecided/mysterious part of the whole is the query parser. Now, Koha support several syntaxes thanks to ZOOM yaz client: PQF, CCL and CQL. Queries in those syntaxes are directly given to ZOOM. I can't figure out how it can be reproduced with other search engine than Zebra... This isn't a small piece of engineering. See above the citation about Data::SearcEngine::ElasticSearch. It's one thing to abstract a search result and its paging, and another thing to abstract a query language--imagine three languages... To be continued... -- Frédéric DEMIANS http://www.tamil.fr/u/fdemians.html
On Sat, Mar 31, 2012 at 05:22:04PM +0200, Frédéric Demians wrote:
For me, the most undecided/mysterious part of the whole is the query parser. Now, Koha support several syntaxes thanks to ZOOM yaz client: PQF, CCL and CQL. Queries in those syntaxes are directly given to ZOOM. I can't figure out how it can be reproduced with other search engine than Zebra... This isn't a small piece of engineering. See above the citation about Data::SearcEngine::ElasticSearch. It's one thing to abstract a search result and its paging, and another thing to abstract a query language--imagine three languages...
grep ZOOM::Query C4/* indeed shows that we are using all variations of ZOOM query languages ;-) But, it seems that rest of Koha code is calling mostly C4::Search::SimpleSearch with small subset of query language which could be rewritten for other engines. I would opt to extend C4::Search API with small special purpose wrappers for various search cases, for example: C4::Search::SimpleSearchCatalog would add "kw,wrdl:$query" for Zebra and pass it to C4::Search::SimpleSearch. This would make $query in code to be only user entered query as opposed to CCL+user query. Problem with our current CCL+user query approach manifests itself in intranet: 1. do search for "human" 2. click Z39.50 Search 3. wonder why Title: includes "kw.wrdl: human" instead of only "human" Clicking Search without erasing first "kw,wrdl:" results in "Nothing found." which is wrong. p.s. is there easy way to get started with SOLR code to test it? I know this was probably answered in Marsielle, but I was too busy with plack ;-) -- Dobrica Pavlinusic 2share!2flame dpavlin@rot13.org Unix addict. Internet consultant. http://www.rot13.org/~dpavlin
Hie, I've worked a lot on Zebra configuration and behavior on two projets at Progilone. I'd like to continue working on it, as personnal contribution. Note that the Lucene search engine used by Solr has a Perl implementation. It is in Apache incubator, so still in havy development : http://incubator.apache.org/projects/lucy.html -- Fridolyn SOMERS fridolyn.somers@gmail.com Marsillargues - France <fridolyn.somers@gmail.com> On Fri, Mar 30, 2012 at 6:15 PM, Paul Poulain <paul.poulain@biblibre.com>wrote:
Hello all,
As you know, our main goal for the oct12 release of Koha is to introduce solr as an alternate search engine. BibLibre already explained which improvements will be added by this search engine on the blog page: http://drupal.biblibre.com/en/blog/entry/solr-developments-for-koha
During the hackfest in Marseille, a group of 4 persons (Claire, Henri-Damien, Juan and Zeno) worked on how this work should be done to be introduced smoothly. The first goal being that a library wanting to run zebra still could. As some librarians could want to use another search engine than zebra or solr, we want to follow a path that would result in a better modularity. I also think that most of us agree that current search code is ugly & very hard to maintain/improve.
The hackfesters have produced a drawing explaining how we could name the different packages:
https://docs.google.com/a/biblibre.com/drawings/d/1ZdsQsoThYgIVSgH3LqgRZy17x... , with a page on the wiki: http://wiki.koha-community.org/wiki/Switch_to_Solr_RFC#.23kohahack12
In this drawing (read from bottom to top), there are 2 main layers "Search" and "Index", that are reponsible of doing searches and doing indexing. The "Conf" object will be responsible to retrieve the configuration (current getIndexes), the "Query" object would be responsible to build the query in SearchEngine grammar, the "Plugin" object would be reponsible to deal with records before indexing (like normalizing data)
Claire (from BibLibre) made a first implementation of this organization on github: https://github.com/clrh/wip-searchengine-layer/tree/master/lib/SearchEngine . Juan (from xercode), also worked on this organization, on the zebra side. His code is available also on github: https://github.com/xercode/Data-SearchEngine-Zebra. Now, Henri-Damien is continuing the work for implementing zebra with this global structure.
In the meantime, 2 other directions have been followed: * Frédéric (Demians, from Tamil) wrote a daemon for zebra indexing (see http://git.tamil.fr/?p=Koha-Contrib-Tamil;a=summary), that resulted in bug http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7759, that document how to introduce this daemon for indexing. Liz (and maybe others) are using it without any problem. This git repository introduces some other tools, but what they effectively do is not completely clear to me (Frédéric, if you want, to add some info...) * Galen (Charlton, from Equinox) wrote some code that you can see in http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7818 and is in "needs signoff" status. The description of the bug includes a lot of things: DOM indexing for biblios (and a tool to automatically write the DOM xsl from the record.abs), and a normalizer for datas, an indexer (Koha::Indexer). Unless I've missed something (Galen, tell if I'm wrong): for now, only the DOM indexing is submitted, normalizer and indexer are not.
What we all agree about: we should have a clearer way to: Normalize / Index / Search in Koha. That's great !
The structure described by the hackfester is great because it's independent from the SearchEngine you use. I think large portions (if not all) of Koha::Contrib::Tamil could be used to write the zebra indexing layer. I also think that The DOM indexing part of what Galen has submitted can be signed-off & pushed without any risk, but the normalize and indexer parts will need coordination to avoid having BibLibre/xercode working in a direction, and Galen working in another. I really like the idea of having normalizer not necessary being MARC; that could be useful in the future.
That's why I propose to organize an IRC meeting (date and time to define, but that will be in Europe afternoon / US morning) with all volunteers to coordinate their efforts. I think this meeting should be regular (monthly ?) After each meeting, a summary of the conclusions would be made on the wiki and posted on this mailing-list.
My proposition: if you're interested by participating to this effort, please answer to this mail. (I'll then start a doodle to find a proper time. I propose 2 hours for the duration of the 1st meeting, then, hopefully, shorter meetings) -Juan/Galen/Zeno, you're considered as being interested by this topic ;-) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
On Wed, Apr 04, 2012 at 09:23:37AM +0200, Fridolyn SOMERS wrote:
Note that the Lucene search engine used by Solr has a Perl implementation. It is in Apache incubator, so still in havy development : http://incubator.apache.org/projects/lucy.html
I'm also a great Lucy fan, and I allmost volonteered to port new search code to Lucy at hackfest. I would love to colaborate with you on this task if you are interested. -- Dobrica Pavlinusic 2share!2flame dpavlin@rot13.org Unix addict. Internet consultant. http://www.rot13.org/~dpavlin
Hie, I think it could be a good maner to keep a light version of Koha search engine. On Wed, Apr 4, 2012 at 3:33 PM, Dobrica Pavlinusic <dpavlin@rot13.org>wrote:
On Wed, Apr 04, 2012 at 09:23:37AM +0200, Fridolyn SOMERS wrote:
Note that the Lucene search engine used by Solr has a Perl implementation. It is in Apache incubator, so still in havy development : http://incubator.apache.org/projects/lucy.html
I'm also a great Lucy fan, and I allmost volonteered to port new search code to Lucy at hackfest. I would love to colaborate with you on this task if you are interested.
-- Dobrica Pavlinusic 2share!2flame dpavlin@rot13.org Unix addict. Internet consultant. http://www.rot13.org/~dpavlin
-- Fridolyn SOMERS fridolyn.somers@gmail.com Marsillargues - France <fridolyn.somers@gmail.com>
Hi to all, Il 30/03/2012 18:15, Paul Poulain ha scritto:
As you know, our main goal for the oct12 release of Koha is to introduce solr as an alternate search engine.
I confim my interest on this workgroup.. For searching language: I think we can support a type of CCLL fro Zebra and Solr, with the syntax of the present opac forms. And with a 'transaltor' to an 'human' for for display. Bye Zeno Tajoli -- Dott. Zeno Tajoli tajoliAT_SPAM_no_prendiATcilea.it fax +39 02 2135520 CILEA - Consorzio Interuniversitario http://www.cilea.it/disclaimer
I agree, CCL for all search engines. Some PQF queries exiting in Koha (in authorities search for example) can easily be converted in CCL. On Thu, Apr 5, 2012 at 5:52 PM, Zeno Tajoli <tajoli@cilea.it> wrote:
Hi to all,
Il 30/03/2012 18:15, Paul Poulain ha scritto:
As you know, our main goal for the oct12 release of Koha is to introduce solr as an alternate search engine.
I confim my interest on this workgroup.. For searching language: I think we can support a type of CCLL fro Zebra and Solr, with the syntax of the present opac forms. And with a 'transaltor' to an 'human' for for display.
Bye Zeno Tajoli -- Dott. Zeno Tajoli tajoliAT_SPAM_no_prendiATcilea.it fax +39 02 2135520 CILEA - Consorzio Interuniversitario http://www.cilea.it/disclaimer _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolyn SOMERS fridolyn.somers@gmail.com Marsillargues - France <fridolyn.somers@gmail.com>
I disagree. As it stands now, CCL is not an acceptable alternative to PQF. CCL can be ambiguous, and runs into all sorts of problems with quoting, etc. The only way I can see a CCL-style query language being a viable option for all searching would be if Koha had a new search parser that was well-defined and turned CCL into something unambiguous (either an internal parser, or, better, an already-existing package that handles all the cases that we'd be likely to throw at it... I don't know what all the options are here, though I know there are at least Search::Query::Parser, Preempt::query::parser, and KinoSearch::Search::QueryParser). Also, of course, losing pqf= as an option for queries would be a pretty large regression for Zebra-based installations. Regards, Jared 2012/4/6 Fridolyn SOMERS <fridolyn.somers@gmail.com>
I agree,
CCL for all search engines. Some PQF queries exiting in Koha (in authorities search for example) can easily be converted in CCL.
On Thu, Apr 5, 2012 at 5:52 PM, Zeno Tajoli <tajoli@cilea.it> wrote:
Hi to all,
Il 30/03/2012 18:15, Paul Poulain ha scritto:
As you know, our main goal for the oct12 release of Koha is to introduce solr as an alternate search engine.
I confim my interest on this workgroup.. For searching language: I think we can support a type of CCLL fro Zebra and Solr, with the syntax of the present opac forms. And with a 'transaltor' to an 'human' for for display.
Bye Zeno Tajoli -- Dott. Zeno Tajoli tajoliAT_SPAM_no_prendiATcilea.it fax +39 02 2135520 CILEA - Consorzio Interuniversitario http://www.cilea.it/disclaimer _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolyn SOMERS fridolyn.somers@gmail.com Marsillargues - France <fridolyn.somers@gmail.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Jared Camins-Esakov Bibliographer, C & P Bibliography Services, LLC (phone) +1 (917) 727-3445 (e-mail) jcamins@cpbibliography.com (web) http://www.cpbibliography.com/
Hi, Il 06/04/2012 13:27, Jared Camins-Esakov ha scritto:
I disagree. As it stands now, CCL is not an acceptable alternative to PQF. CCL can be ambiguous, and runs into all sorts of problems with quoting, etc.
Probaly not a pure CCL but something very similar to the present opac queries that are so: idx=ti,phr &q=red+winter &op=and &idx=au,wrdl &q=white &op=and &idx=kw &op=and &idx=kw &sort_by=call_number_asc&do=Search Bye -- Dott. Zeno Tajoli tajoliAT_SPAM_no_prendiATcilea.it fax +39 02 2135520 CILEA - Consorzio Interuniversitario http://www.cilea.it/disclaimer
Zeno,
Probaly not a pure CCL but something very similar to the present opac queries that are so:
idx=ti,phr &q=red+winter &op=and
&idx=au,wrdl &q=white &op=and
&idx=kw &op=and
&idx=kw
&sort_by=call_number_asc&do=Search
If we're using that we could just as easily use PQF internally, with a lookup table to convert indexes to bib-1 attributes. Also, to clarify for those who might not be familiar with the terms, CCL is ISO 8777 ( http://www.indexdata.com/yaz/doc/tools.html#CCL ). It is a standard for specifying queries in a single, non-tokenized string. So this would be an example CCL query:
ti,phr:"red winter" and au,wrdl:white
Notice how ambiguous that query becomes if you enter without the quotes (which users often don't enter):
ti,phr:red winter and au,wrdl:white
Where should the system be looking for the word "winter"? Consider this even more charming example:
ti:snow white and rose red and au:wrede
I'm not sure any commentary is necessary, but the book I'm looking for with that query is actually entitled "Snow White and Rose Red" by Patricia C. Wrede. If we use CCL, this record *may* match that search: TITLE: Snow white AUTHOR: Wrede, Patricia C. PUBLISHER:Rose red productions In order to handle real-life input, Koha uses an implied keyword index. Much better to use a tokenized query and convert it internally into equally non-ambiguous syntaxes handled by Zebra and Solr. Regards, Jared -- Jared Camins-Esakov Bibliographer, C & P Bibliography Services, LLC (phone) +1 (917) 727-3445 (e-mail) jcamins@cpbibliography.com (web) http://www.cpbibliography.com/
participants (6)
-
Dobrica Pavlinusic -
Fridolyn SOMERS -
Frédéric Demians -
Jared Camins-Esakov -
Paul Poulain -
Zeno Tajoli