Search Engine Changes : let's get some solr
Hi As you already read in Paul previous message about "BibLibre strategy for 3.4 and next version", we are growing, want be involved in the community as previously. Paul promised some POCs, here is one available. We also worked on Plack and support. We created a base of script to search for Memoryleaks. We'll demonstrate that later. zebra is fast and embeds native z3950 server. But it has also some major drawbacks we have to cope with on our everyday life making it quite difficult to maintain. 1. zebra config files are a nightmare. You can't drive the configuration file easily. namely : Can't edit indexs via HTTP or configuration. all is in files hardcoded on disk. ⇒ you can't list indexes you can't change indexes, you can't edit indexes, you can't say I want this index at OPAC, that in intranet. (Could be done with scraping ccl.properties, and then record.abs and bib1.att…. But what a HELL) So you cannot customize configuration defining the indexes you want easily. And ppl donot get a translation of the indexes since all the indexes are hardcoded in the ccl.properties and we donot have a translation process so that ccl attributes could be translated into different languages. 2. no real-time indexing : the use of a crontab is poor: when you add an authority while creating a biblio, you have to wait some some minutes to end your biblio (might be solved since zebra has some way to index biblios via z3950 extended services, but hard and should be tested and at the time community first tested that, a performance problem was raised on indexing.) 3. no way to access/process/delete data easily. If you have indexes in it or have some problems with your data, you have to reindex the whole stuff and indexing errors are quite difficult to detect. 4. during index process of a file, if you have a problem in your data, zebraidx just fails silently… And this is NOT secure. And you have no way to know WHICH biblio made the process crash. We had a LOT of trouble with Aix-Marseille universities that have some arabic translitterated biblios that makes zebra/icu completly crash ! We had to do some recursive script to find 14 biblios on 730 000 that makes zebra crash (even is properly stored & displayed) 5. facets are not working properly : they are on the result displayed because there are problems with diacritics & facets that can't be solved as of today. And noone can provide a solution (we spoke about that with indexdata and no clear solution was really provided. 6. zebra does not evolve anymore. There is no real community around it, it's just an opensource indexdata software. We sent many questions onlist and never got answers. We could pay for better support but the fee required is quite deterrent and benefit is still questionable. 7. icu & zebra are colleagues, not really friends : right truncation not working, fuzzy search not working and facets. 8. we use a deprecated way to define indexes for biblios (grs1) and the tool developped by indexdata to change to DOM has many flaws. we could manage and do with it. But is it worth the strive ? I think that every one agrees that we have to refactor C4::Search. Indeed, query parser is not able to manage independantly all the configuration options. And usage of usmarc as internal for biblio comes with a serious limitation of 9999 bytes, which for big biblios with many items, is not enough. BibLibre investigated in a catalogue based on solr. A University in France contracted us for that development. This University is in relation with all the community here in France and solr will certainly be adopted by all the libraries France wide. We are planning to release the code on our git early spring next year and rebase on whatever Koha version will be released at that time 3.4 or 3.6. Why ? Solr indexes with data with HTTP. It can provide fuzzy search, search on synonyms, suggestions It can provide facet search, stemming. utf8 support is embedded. Community is really impressively reactive and numerous and efficient. And documentation is very good and exhaustive. You can see the results on solr.biblibre.com and catalogue.solr.biblibre.com http://catalogue.solr.biblibre.com/cgi-bin/koha/opac-search.pl?q=jean http://solr.biblibre.com/cgi-bin/koha/admin/admin-home.pl you can log there with demo/demo lgoin/password http://solr.biblibre.com/cgi-bin/koha/solr/indexes.pl is the page where ppl can manage their indexes and links. a) Librarians can define their own indexes, and there is a plugin that fetches data from rejected authorities and from authorised_values (that could/should have been achieved with zebra but only with major work on xslt). b) C4/Search.pm count lines of code could be shrinked ten times. You can test from poc_solr branch on git://git.biblibre.com/koha_biblibre.git But you have to install solr. Any feedback/idea welcome. -- Henri-Damien LAURENT BibLibre
Hi to all,
Any feedback/idea welcome.
the main problem that I see is that Zebra is much more light on RAM. Solrs is needs Java + an App server J2EE (Tomact, Jetty ?). If we select Solr, can we setup a library with 512 MB of RAM in the server ? Bye -- Zeno Tajoli CILEA - Segrate (MI) tajoliAT_SPAM_no_prendiATcilea.it (Indirizzo mascherato anti-spam; sostituisci qaunto tra AT con @)
Hi, Maybe a possible alternative to Lucene with Java could be CLucene (http://clucene.sourceforge.net/) to increase performance. But seems to be in a less mature stage that the Java brother and is only the library behind the search engine. Have a Cpan module: http://search.cpan.org/~tbusch/Lucene-0.18/lib/Lucene.pm Salva El 04/10/2010 11:04, Zeno Tajoli escribió:
Hi to all,
Any feedback/idea welcome.
the main problem that I see is that Zebra is much more light on RAM. Solrs is needs Java + an App server J2EE (Tomact, Jetty ?). If we select Solr, can we setup a library with 512 MB of RAM in the server ?
Bye
On 04/10/10 10:30, Salvador Zaragoza Rubio wrote:
Hi,
Maybe a possible alternative to Lucene with Java could be CLucene (http://clucene.sourceforge.net/) to increase performance. But seems to be in a less mature stage that the Java brother and is only the library behind the search engine. Have a Cpan module: http://search.cpan.org/~tbusch/Lucene-0.18/lib/Lucene.pm
If looking at cpan modules take a look at KinoSearch as well. -- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 208 366 1295 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2 http://www.ptfs-europe.com
Le 04/10/2010 11:04, Zeno Tajoli a écrit :
Hi to all,
Any feedback/idea welcome.
the main problem that I see is that Zebra is much more light on RAM. Solrs is needs Java + an App server J2EE (Tomact, Jetty ?). If we select Solr, can we setup a library with 512 MB of RAM in the server ?
Bye
Looking at most users we know, the problem of little RAM is the least of their problems, the question about accurate facetting with correct encoding and the ease of use, add and display index is more of their concerns. So yes, problem of RAM, but it could be overcome with a common web service say on solr.koha-community.org for instance, people could provide on an external server to index and get search results from. We chose to base our POC on Data::SearchEngine If you want to know more details on dependencies. -- Henri-Damien LAURENT
Wouldn't using Solr also give us the flexibility to use non-MARC metadata schemas, like MODS, METS, Dublin Core or EAD? Indexes could be defined for the whole system, with mappings for each supported metadata scheme on how to get the data into said indexes. The increased flexibility seems like it would be worth the higher system requirements, at least for folks who could afford a suitably powerful server. The installation process should provide options to help tune Koha to the available hardware (light and fast OR heavier and more powerful). Cheers, -Ian On Mon, Oct 4, 2010 at 6:38 AM, LAURENT Henri-Damien <laurenthdl@alinto.com>wrote:
Le 04/10/2010 11:04, Zeno Tajoli a écrit :
Hi to all,
Any feedback/idea welcome.
the main problem that I see is that Zebra is much more light on RAM. Solrs is needs Java + an App server J2EE (Tomact, Jetty ?). If we select Solr, can we setup a library with 512 MB of RAM in the server ?
Bye
Looking at most users we know, the problem of little RAM is the least of their problems, the question about accurate facetting with correct encoding and the ease of use, add and display index is more of their concerns. So yes, problem of RAM, but it could be overcome with a common web service say on solr.koha-community.org for instance, people could provide on an external server to index and get search results from.
We chose to base our POC on Data::SearchEngine If you want to know more details on dependencies.
-- Henri-Damien LAURENT _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
-- Ian Walls Lead Development Specialist ByWater Solutions Phone # (888) 900-8944 http://bywatersolutions.com ian.walls@bywatersolutions.com Twitter: @sekjal
Le 04/10/2010 12:38, LAURENT Henri-Damien a écrit :
Le 04/10/2010 11:04, Zeno Tajoli a écrit :
Hi to all,
Any feedback/idea welcome.
the main problem that I see is that Zebra is much more light on RAM. Solrs is needs Java + an App server J2EE (Tomact, Jetty ?). If we select Solr, can we setup a library with 512 MB of RAM in the server ?
Bye
Looking at most users we know, the problem of little RAM is the least of their problems, the question about accurate facetting with correct encoding and the ease of use, add and display index is more of their concerns. So yes, problem of RAM, but it could be overcome with a common web service say on solr.koha-community.org for instance, people could provide on an external server to index and get search results from.
We chose to base our POC on Data::SearchEngine If you want to know more details on dependencies.
Just as a side note, zebra3 should be based on solr. So yet another point for solr.
Just as a side note, zebra3 should be based on solr. So yet another point for solr.
So, isn't it the best interest of the Koha community to wait that Indexdata integrate Solr into Zebra? and in the meantime, switch Koha-Zebra interface from GRS1 to Dom in order to gain in granularity and customazibility. And as suggested by Thomas Dukleth, Koha supporters could sponsor directly Indexdata to speed up their work and give them a Koha 'direction'. Who's better than Indexdata has deep knowledge of search engine technlogy in Libraries arena? and could conciliate and accommodate Solr at best to the library's needs and peculiarities? -- Frédéric
Frederic Demians writes
Just as a side note, zebra3 should be based on solr. So yet another point for solr.
So, isn't it the best interest of the Koha community to wait that Indexdata integrate Solr into Zebra? and in the meantime, switch Koha-Zebra interface from GRS1 to Dom in order to gain in granularity and customazibility.
Yes. In any case, the zebra documentation notes that GRS1 has been improved and replaced by DOM.
And as suggested by Thomas Dukleth, Koha supporters could sponsor directly Indexdata to speed up their work and give them a Koha 'direction'. Who's better than Indexdata has deep knowledge of search engine technlogy in Libraries arena?
And to appear credible to them, it wouldn't be best to adopt their recent technologies, before asking them to develop new ones? Cheers, Thomas Krichel http://openlib.org/home/krichel http://authorclaim.org/profile/pkr1 skype: thomaskrichel
On Tue, Oct 12, 2010 at 8:41 AM, Thomas Krichel <krichel@openlib.org> wrote:
Yes. In any case, the zebra documentation notes that GRS1 has been improved and replaced by DOM.
And as suggested by Thomas Dukleth, Koha supporters could sponsor directly Indexdata to speed up their work and give them a Koha 'direction'. Who's better than Indexdata has deep knowledge of search engine technlogy in Libraries arena?
And to appear credible to them, it wouldn't be best to adopt their recent technologies, before asking them to develop new ones?
When running Koha's Makefile.PL, DOM is the default choice for zebra and has been for quite some time. Kind Regards, Chris
Christopher Nighswonger writes
When running Koha's Makefile.PL, DOM is the default choice for zebra and has been for quite some time.
Oh good because from reading the comments here I got the impression that it was not. Cheers, Thomas Krichel http://openlib.org/home/krichel http://authorclaim.org/profile/pkr1 skype: thomaskrichel
Le 12/10/2010 14:47, Christopher Nighswonger a écrit :
On Tue, Oct 12, 2010 at 8:41 AM, Thomas Krichel <krichel@openlib.org> wrote:
Yes. In any case, the zebra documentation notes that GRS1 has been improved and replaced by DOM.
And as suggested by Thomas Dukleth, Koha supporters could sponsor directly Indexdata to speed up their work and give them a Koha 'direction'. Who's better than Indexdata has deep knowledge of search engine technlogy in Libraries arena?
And to appear credible to them, it wouldn't be best to adopt their recent technologies, before asking them to develop new ones?
When running Koha's Makefile.PL, DOM is the default choice for zebra and has been for quite some time.
Kind Regards, Well, actually DOM has been default choice for zebra Authority server. For biblio server, it still is GRS1 as far as I know. Friendly. -- Henri-Damien LAURENT
Le 12/10/10 14:47, Christopher Nighswonger a écrit :
When running Koha's Makefile.PL, DOM is the default choice for zebra and has been for quite some time.
It isn't available for UNIMARC which use GRS1 for biblio and authorities records. For MARC21, DOM is used only for authorities records. Biblio records still use GRS1 Record Model. In theory--and I understand Henri-Damien Laurent frustration to conciliate theory and practice with Zebra--DOM Record Model should allow to do a lot of things required by Koha via successive XSLT transformations: strip out title leading articles marked by NSB/NSE characters for Title sort index; Unicode normalization; facets (?)... It'll be a great pity for Index Data to loose Koha. We need help from Index Data to see if it's possible to keep Zebra into Koha... -- Frédéric
Adding to the aforementioned limitations and desired improvements, another issue with Zebra is that it index builds scale very poorly. Indexes must be built serially, and it's a really problematic limitation for large catalogs. A parallelized index creation process would be a huge benefit. Does SOLR have that capability? Also, what do you have in mind for continuing Z39.50 support? This is a must-have feature for many libraries. I've been investigating the possibility of using MongoDB or a similar dynamic indexer to replace Zebra, but the need to write a Z39.50 front end adds a great deal more work to the project. Clay On Mon, Oct 4, 2010 at 1:10 AM, LAURENT Henri-Damien < henridamien.laurent@biblibre.com> wrote:
Hi As you already read in Paul previous message about "BibLibre strategy for 3.4 and next version", we are growing, want be involved in the community as previously. Paul promised some POCs, here is one available. We also worked on Plack and support. We created a base of script to search for Memoryleaks. We'll demonstrate that later.
zebra is fast and embeds native z3950 server. But it has also some major drawbacks we have to cope with on our everyday life making it quite difficult to maintain.
1. zebra config files are a nightmare. You can't drive the configuration file easily. namely : Can't edit indexs via HTTP or configuration. all is in files hardcoded on disk. => you can't list indexes you can't change indexes, you can't edit indexes, you can't say I want this index at OPAC, that in intranet. (Could be done with scraping ccl.properties, and then record.abs and bib1.att.... But what a HELL) So you cannot customize configuration defining the indexes you want easily. And ppl donot get a translation of the indexes since all the indexes are hardcoded in the ccl.properties and we donot have a translation process so that ccl attributes could be translated into different languages.
2. no real-time indexing : the use of a crontab is poor: when you add an authority while creating a biblio, you have to wait some some minutes to end your biblio (might be solved since zebra has some way to index biblios via z3950 extended services, but hard and should be tested and at the time community first tested that, a performance problem was raised on indexing.)
3. no way to access/process/delete data easily. If you have indexes in it or have some problems with your data, you have to reindex the whole stuff and indexing errors are quite difficult to detect.
4. during index process of a file, if you have a problem in your data, zebraidx just fails silently... And this is NOT secure. And you have no way to know WHICH biblio made the process crash. We had a LOT of trouble with Aix-Marseille universities that have some arabic translitterated biblios that makes zebra/icu completly crash ! We had to do some recursive script to find 14 biblios on 730 000 that makes zebra crash (even is properly stored & displayed)
5. facets are not working properly : they are on the result displayed because there are problems with diacritics & facets that can't be solved as of today. And noone can provide a solution (we spoke about that with indexdata and no clear solution was really provided.
6. zebra does not evolve anymore. There is no real community around it, it's just an opensource indexdata software. We sent many questions onlist and never got answers. We could pay for better support but the fee required is quite deterrent and benefit is still questionable.
7. icu & zebra are colleagues, not really friends : right truncation not working, fuzzy search not working and facets.
8. we use a deprecated way to define indexes for biblios (grs1) and the tool developped by indexdata to change to DOM has many flaws. we could manage and do with it. But is it worth the strive ?
I think that every one agrees that we have to refactor C4::Search. Indeed, query parser is not able to manage independantly all the configuration options. And usage of usmarc as internal for biblio comes with a serious limitation of 9999 bytes, which for big biblios with many items, is not enough.
BibLibre investigated in a catalogue based on solr. A University in France contracted us for that development. This University is in relation with all the community here in France and solr will certainly be adopted by all the libraries France wide. We are planning to release the code on our git early spring next year and rebase on whatever Koha version will be released at that time 3.4 or 3.6.
Why ?
Solr indexes with data with HTTP. It can provide fuzzy search, search on synonyms, suggestions It can provide facet search, stemming. utf8 support is embedded. Community is really impressively reactive and numerous and efficient. And documentation is very good and exhaustive.
You can see the results on solr.biblibre.com and catalogue.solr.biblibre.com
http://catalogue.solr.biblibre.com/cgi-bin/koha/opac-search.pl?q=jean http://solr.biblibre.com/cgi-bin/koha/admin/admin-home.pl you can log there with demo/demo lgoin/password
http://solr.biblibre.com/cgi-bin/koha/solr/indexes.pl is the page where ppl can manage their indexes and links.
a) Librarians can define their own indexes, and there is a plugin that fetches data from rejected authorities and from authorised_values (that could/should have been achieved with zebra but only with major work on xslt).
b) C4/Search.pm count lines of code could be shrinked ten times. You can test from poc_solr branch on git://git.biblibre.com/koha_biblibre.git But you have to install solr.
Any feedback/idea welcome. -- Henri-Damien LAURENT BibLibre _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
Le 04/10/2010 22:16, Fouts, Clay a écrit :
Adding to the aforementioned limitations and desired improvements, another issue with Zebra is that it index builds scale very poorly. Indexes must be built serially, and it's a really problematic limitation for large catalogs. A parallelized index creation process would be a huge benefit. Does SOLR have that capability?
Also, what do you have in mind for continuing Z39.50 support? This is a must-have feature for many libraries. I've been investigating the possibility of using MongoDB or a similar dynamic indexer to replace Zebra, but the need to write a Z39.50 front end adds a great deal more work to the project.
Hi Clay. Serious questions. About z3950 support, I investigated in JZ3950 Kit http://k-int.blogspot.com/2008/05/exposing-solr-services-as-z3950-server.htm... It seems a serious candidate. About parallelized index creation process, I think I read something about that for solr 1.4.1. But not sure if I can grab on that soon. At least we are investigating solr multicore in order to use solr at its best. -- Henri-Damien LAURENT
On this subject, some feedback would be welcomed from members of the Koha community who are also members from Evergreen community. As said by others, small to medium-sized libraries shouldn't be scarified to large libraries needs. A debian-ubuntu Koha package, installable in one-click, seems now an achievable goal. New search engine integration into Koha shouldn't distract from this goal. -- Frédéric DEMIANS
1. Z39.50/SRU SUPPORT. If I would have any specific concern about the prospect of new development complicating long term development, it would be about the possibility of breaking or neglecting necessary Z39.50/SRU server support in the process of adding excessively generic Solr/Lucene indexing. Z39.50/SRU are important library standards for record sharing which is vital to the good functioning of the library community. I commend Henri-Damien Laurent for taking the issue of Z39.50/SRU support seriously and finding JZKit as a possible solution for Z39.50/SRU support using Solr/Lucene. 2. AVOIDING FEATURE REGRESSION OR BLOCKS TO FUTURE DEVELOPMENT. Popular implementations of Solr/Lucene in library automation systems have made all the mistakes of sacrificing the precision needed for serious library research in return for high recall with poor relevancy often found in Google which may merely satisfy casual queries. I share the concern that working with Zebra is too much like working with a black box into which one cannot peer. I make no claim that existing Z39.50/SRU Zebra support in Koha is ideal but merely than it should not be too easily sacrificed for something else with its own problems which are merely less familiar to us. I suggest that we retain the existing Z39.50/SRU Zebra support in Koha while adding other options which may improve local indexing. The full use of Bib-1 position, structure, and completeness attributes for Z39.50 or the ordered prox CQL operator for SRU would allow the precise queries needed for serious research. The lack of a completeness operator in CQL is a serious deficiency for SRU. Index Data may still need to develop support in Zebra for the ordered prox CQL operator which will most likely require paying to support that effort when it would be appreciated in the Koha community. Zebra certainly has bugs as does all software. [See the end of the document for the Index Data promise about bugs.] Ultimately, I see no manageable way to have a free software library automation system without paying for some support for something from Index Data even if that would merely be Z39.50/SRU client programming libraries. Solr/Lucene may now be a good choice for internal indexing in Koha. Lucene was not considered fairly during 2005 testing for Koha because the Perl bindings at that time were notoriously slow. Solr and Lucene have long had the mind share and development advantage of being Apache Foundation projects which Zebra will never match, hence the forthcoming inclusion of Solr/Lucene indexing for the next major versions of Pazpar2 and Zebra . However, Solr/Lucene has had problems which should not go unconsidered in evaluating or actually implementing Solr/Lucene based indexing in Koha. I am not certain what point 8 from Henri-Damien's message is specifically meant to criticise. Is the complaint against indexing based the DOM in general or against the frustration of needing to migrate from an inefficient deprecated means of using the DOM to a more efficient means of using it? On Mon, October 4, 2010 08:10, LAURENT Henri-Damien wrote: [...]
8. we use a deprecated way to define indexes for biblios (grs1) and the tool developped by indexdata to change to DOM has many flaws. we could manage and do with it. But is it worth the strive ?
[...] I contend that although working with the DOM can be difficult at times, the DOM helps provide needed flexibility and precision in indexing. 2.1. HISTORICAL LACK OF PRECISION IN SOLR/LUCENE. Solr/Lucene may have been a poor choice during the 2004 - 2006 period of sponsoring Perl Zoom and developing Zebra in Koha. Lucene had originally been developed for full text indexing of unstructured documents. Solr had originally been merely an easy to configure front end to a subset of Lucene functionality. Solr became a popular choice for the simplest free software OPACs. I have always tried to subject choices taken in Koha to personal reconsideration and made a modest investigation of the capabilities of Lucene and Solr/Lucene in 2007. I consulted widely and attended some conferences asking questions of the most expert implementers of library automation systems who had been using Lucene or Solr/Lucene. I tried to consult with people working to solve real problems rather than merely relying upon possibly incomplete documentation. In 2007, Solr provided no support for indexing to serve important concepts used for obtaining precision in library systems. 2.1.1. ASPECTS OF PRECISION HISTORICALLY UNSUPPORTED BY SOLR/LUCENE. Hierarchy where some content is subsidiary to other content and content derives meaning from the place in the hierarchy had no support in Solr circa 2007. Field to subfield relationships is an example of hierarchy in MARC records. Namespace hierarchies are examples of hierarchy in XML records and are accessible by XPath queries. Hierarchy is a fundamental feature of classification and retrieval for easily including wanted record sets and excluding unwanted record sets. Sequential order where the order of separate record sub-elements is relevant to meaning had no support in Solr circa 2007. Philosophy - History, meaning 'history of philosophy', is an entirely different subject from History - Philosophy, meaning 'philosophy of history'. Note that the inversion of word order between individual controlled vocabulary elements and the corresponding English phrase with the same meaning. The sequential order of fields within a record or MARC subfields within a particular field are examples of sequential order in MARC records. The sequential order of namespaces within a record and the order of repeated elements within the same namespace are examples of sequential order in XML records accessible by XPath queries. Sequential order is a fundamental feature of meaning in language and is not necessarily reducible to phrase strings where interceding terms may or may not be present and word order may be inverted as in the example given. 2.1.2. ALTERNATIVES FOR PRECISION USING LUCENE. In 2005, work at Bibliothèque de l'Université Laval (originators of RAMEAU) had developed LIUS (Lucene Index Update and Search) to overcome some difficulties of Lucene including fielded indexing of the very simplest flat field metadata found in some general purpose document types and XPath indexing for XML documents, http://sourceforge.net/projects/lius/ . Laval now uses Solr/Lucene based Constellio, http://www.constellio.com/ . In 2007, I had been informed by a programmer of library automation systems working in the pharmaceutical industry, if I remember his job correctly, that hierarchical indexing and sequential indexing could be done in Lucene but that there was no support for such indexing in Solr. Precision is very important for both scientific and business purposes in the pharmaceutical industry. Despite valid criticism of some business practises within the pharmaceutical industry, lives are often at stake in their work. We should treat the quality of information retrieval in library automation systems as if lives are at stake. Lives will sometimes be at stake in the research which people do. 2.2. CONSEQUENCES OF LACK OF PRECISION. Sadly, the concept of precision has not been one which signified in the minds of those developing the popular free software OPACs using Solr/Lucene or some of their non-free equivalents. Examples of the consequences to which Koha is not excluded are using only $a in faceting despite the presence of other important subfields; jumbling all the subfields from all similar fields independently; and returning irrelevant results because subfields have been treated as mere independent keywords devoid of contextual meaning even in the context of a query using an authority controlled field. Human nature, to which Koha is not immune, may have some impetus to oversimplify for an expected advantage. Oversimplification in the context of a library automation system could eliminate the ability for the user to access the real complexity and richness of relationships in bibliographic records to improve speed or robustness. Such oversimplification exists to a large extent in every actual library automation system. I may be raising a false alarm about the possibility that some feature advance may complicate or block better improvements in the future. Yet, I prefer to take a vigilant stance rather than be sorry later for not having raised a concern. 2.3. CURRENT SUITABILITY OF SOLR/LUCENE. I note significant improvements identified in the Solr/Lucene changelog from version 1.3 in 2008 and later. The DataImportHandler was added in version 1.3. DataImportHandler has options for XPath based indexing. Solr still seems to have no support for ordered proximity searches. Perhaps XPath based indexing would address the problem. A possible workaround modifying the Lucene code in SolrQueryParser to return SpanNearQuery instead of PhraseQuery may be a very undesirable remedy, breaking one feature to fix another. Whether the improvements in Solr/Lucene are sufficient to overcome the past limitations which I have identified would require experimentation. 3. SUPPORT MODELS FOR NEEDED PROGRAMMING LIBRARIES. It is good that companies such as Knowledge Integration, http://www.k-int.com/ , developers of JZKit, http://www.k-int.com/jzkit , are providing some free software competition and complementary work to what is available from Index Data. Note that the JZKit developer, Ian Ibbotson, is using Yaz as a Z39.50 client, http://k-int.blogspot.com/2008/05/exposing-solr-services-as-z3950-server.htm... leaving a dependency on Index Data for Z39.50 for client side services. There is some prevarication at Index Data against fully embracing free software in everything they do. Inevitably they need revenue to be sustainable. The following thought about a possible shortage of Index Data development time and the consequences is merely speculative but not uninformed. Index Data may have a problem of not enough developers working for them with sufficient experience to further the development of the underlying programming libraries which we use to meet the amount of the work which the library community hopes to have from them. Contracting for Index Data development in the absence of sufficient development time to go around might result in bidding for the importance of the development which you need as much as it is sharing the cost of development with others. Would working with Knowledge Integration which has even fewer developers be significantly different in terms of development costs? Does Knowledge Integration need less money for a given amount of work than Index Data does to be sustainable? Consider that JZKit seems to have no documentation worthy of identifying as documentation. The source code repository contains about four pages of outlines for documentation with only one sentence of actual content, http://www.k-int.com/developer/downloads . There are some comments in the source code which I understand are used as documentation for JZKit. Yet the comments are too few and incomplete to be of sufficient use to me and from what I have noted others as well. There are some virtually empty example configuration files which could be used as a basis for speculating how configuration works. JZKit supposedly has a mailing list but I have not found it. Index Data does provide documentation even if we have often found it inadequate for our needs in Koha development. I suspect that sufficient documentation at Knowledge Integration as at Index Data requires a support contract and as we know has no guarantees for completeness. Writing clear and thorough documentation is hard work. Writing documentation is the last thing which programmers generally want to do. Lack of good documentation is a common characteristic of free software. If there would also be a need for JZKit to have some missing feature or better functionality, would the situation also be any different for Knowledge Integration development than Index Data development? See the unfortunate position of Knowledge Integration on GPL contributions or AGPL 3 contributions in the case of JZKit, http://www.k-int.com/developer/participate . The library community needs to find the means of working more cooperatively to ensure a steady availability of development resources at companies such as Index Data and Knowledge Integration for sustainable shared development. I hope that Index Data may eventually be won over from their sometimes prevaricative position towards free software. Yet they need to be sustainable by some means. I do not find the position of of Knowledge Integration to be any different and note that they do not have a link to the source code repository for OpenHarvest, http://www.k-int.com/developer/downloads . Index Data does have a long history of supporting free software for libraries. Index Data also makes an extraordinary almost impossible to be believed promise in their support contracts to fix any bug within set number of days. The issue of how to share the cost of support contracts for programming libraries provided by companies such as Index Data or Knowledge Integration across multiple Koha support companies or even outside of the Koha community needs to be considered. [...] Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783
I post this email while away from my work area and access to most of my research materials; more specifically, on a netbook from a hospital room, although I personally am fine. I hope my question is not thus poorly presented. Does the term "precision" have a meaning significantly different when applied to the indexing of databases than it does in general use? Quote: "The precision of a measurement system, also called reproducibility or repeatability, is the degree to which repeated measurements under unchanged conditions show the same results." http://en.wikipedia.org/wiki/Accuracy_and_precision I was not aware that precision was a problem with Zebra, although we have found, to our great dismay with our recent implementation of Koha, that relevancy, which I would call accuracy, is. More correctly and scientifically stated perhaps, relevancy sucks. Do I incorrectly associate accuracy with relevancy? Quote: "...accuracy of a measurement system is the degree of closeness of measurements of a quantity to its actual (true) value." IBID Greg Lawson Rolling Hills Consolidated Library ------------------------------
1. Z39.50/SRU SUPPORT....
Koha community needs to be considered. [...] Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
* glawson@rhcl.org (glawson@rhcl.org) wrote:
I post this email while away from my work area and access to most of my research materials; more specifically, on a netbook from a hospital room, although I personally am fine. I hope my question is not thus poorly presented.
Does the term "precision" have a meaning significantly different when applied to the indexing of databases than it does in general use?
Quote:
"The precision of a measurement system, also called reproducibility or repeatability, is the degree to which repeated measurements under unchanged conditions show the same results." http://en.wikipedia.org/wiki/Accuracy_and_precision
I was not aware that precision was a problem with Zebra, although we have found, to our great dismay with our recent implementation of Koha, that relevancy, which I would call accuracy, is. More correctly and scientifically stated perhaps, relevancy sucks. Do I incorrectly associate accuracy with relevancy?
Greg I have noticed a bug with C4/Search .. such that relevancy isn't actually being used. Is your opac live? If you send me a url I can test my theory on your opac Chris
Quote:
"...accuracy of a measurement system is the degree of closeness of measurements of a quantity to its actual (true) value." IBID
Greg Lawson Rolling Hills Consolidated Library
------------------------------
1. Z39.50/SRU SUPPORT....
Koha community needs to be considered. [...] Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
-- Chris Cormack Catalyst IT Ltd. +64 4 803 2238 PO Box 11-053, Manners St, Wellington 6142, New Zealand
http://opac.rhcl.org I have some really good test data to illustrate my point, but don't have the data here with me. Greg On 10/10/2010 05:37 PM, Chris Cormack wrote:
* glawson@rhcl.org (glawson@rhcl.org) wrote:
I post this email while away from my work area and access to most of my research materials; more specifically, on a netbook from a hospital room, although I personally am fine. I hope my question is not thus poorly presented.
Does the term "precision" have a meaning significantly different when applied to the indexing of databases than it does in general use?
Quote:
"The precision of a measurement system, also called reproducibility or repeatability, is the degree to which repeated measurements under unchanged conditions show the same results." http://en.wikipedia.org/wiki/Accuracy_and_precision
I was not aware that precision was a problem with Zebra, although we have found, to our great dismay with our recent implementation of Koha, that relevancy, which I would call accuracy, is. More correctly and scientifically stated perhaps, relevancy sucks. Do I incorrectly associate accuracy with relevancy?
Greg
I have noticed a bug with C4/Search .. such that relevancy isn't actually being used. Is your opac live? If you send me a url I can test my theory on your opac
Chris
Quote:
"...accuracy of a measurement system is the degree of closeness of measurements of a quantity to its actual (true) value." IBID
Greg Lawson Rolling Hills Consolidated Library
------------------------------
1. Z39.50/SRU SUPPORT....
Koha community needs to be considered. [...] Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
-- Greg Lawson Rolling Hills Consolidated Library 1912 N. Belt Highway St. Joseph, MO 64506
Reply inline: On Sun, October 10, 2010 22:26, glawson@rhcl.org wrote: [...] 1. PRECISION AND BUGS IN KOHA ZEBRA IMPLEMENTATION.
Does the term "precision" have a meaning significantly different when applied to the indexing of databases than it does in general use?
The ordinary language use of precision has basically the same meaning as the more specialist uses which measure precision mathematically.
Quote:
"The precision of a measurement system, also called reproducibility or repeatability, is the degree to which repeated measurements under unchanged conditions show the same results." http://en.wikipedia.org/wiki/Accuracy_and_precision
The measurement theory Wikipedia article is fine. The Wikipedia article treating the same concepts for information retrieval is http://en.wikipedia.org/wiki/Precision_and_recall . Precision has a disambiguation page, http://en.wikipedia.org/wiki/Precision .
I was not aware that precision was a problem with Zebra, although we have found, to our great dismay with our recent implementation of Koha, that relevancy, which I would call accuracy, is. More correctly and scientifically stated perhaps, relevancy sucks.
I did not assert that precision is necessarily a problem with Zebra. Henri-Damien Laurent listed some Zebra bugs which could be construed as affecting precision by not returning a result set or returning a result set in which some multi-byte characters are mangled. Zebra bugs need to be fixed. What I did assert is that we are not yet using some options in Zebra Z39.50 support which allow for better precision matching sets of controlled terms especially useful for subject and classification based queries. There has not been sufficient programming time available to implement the options to which I referred. The underlying Koha implementation of authority control and support for classification needs improvement first. I suspect that the relevancy issues to which you are referring are different and relate to ranking the result set, possibly adding indexes appropriate to your organisation, and using appropriate fielded queries. You would need to state the problematic results which you have for particular queries to know the problem to which you are referring. Chris Cormack may have identified at least one of the problems for you in his reply, http://lists.koha-community.org/pipermail/koha-devel/2010-October/034470.htm... .
Do I incorrectly associate accuracy with relevancy?
I think that you correctly associate accuracy with relevancy. I am not certain where recall would fit in comparing information retrieval terms with measurement theory terms.
Quote:
"...accuracy of a measurement system is the degree of closeness of measurements of a quantity to its actual (true) value." IBID
2. CONSIDERING OPTIONS CAREFULLY. My concern is that we do not completely abandon the Zebra indexing system which we now have because of some bugs which could be fixed. We could have a sophisticated Z39.50/SRU server and Solr/Lucene indexing by working with Index Data and with bugs fixed for a moderate service contract. Given the undocumented but apparently unsophisticated feature set of JZKit, I suspect that it will be more expensive to have Knowledge Integration develop a sufficiently sophisticated Z39.50/SRU server implementation in JZKit. The large Solr/Lucene community may support many of our interests well without needing to depend upon us financially, which would be a real advantage. However, such a non-library community, despite the presence of some library community members, is liable to take a very long time to appreciate the value of sophisticated implementations of precision to support all types of library queries well. [...] Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783
Le 11/10/2010 10:57, Thomas Dukleth a écrit :
My concern is that we do not completely abandon the Zebra indexing system which we now have because of some bugs which could be fixed. We could have a sophisticated Z39.50/SRU server and Solr/Lucene indexing by working with Index Data and with bugs fixed for a moderate service contract. Given the undocumented but apparently unsophisticated feature set of JZKit, I suspect that it will be more expensive to have Knowledge Integration develop a sufficiently sophisticated Z39.50/SRU server implementation in JZKit.
Frankly, The search code in Koha is a nightmare. We try to construct the ccl query, build facets, ... All of this is done directly by Lucene/solR. In our POC, we removed 90% of the lines of Search.pm I'm 100% sure it's much more work to fix all this stuff than rewrite it from scratch. The only thing we must take big care of, imo, will be how to migrate existing zebra-koha libraries to solR-koha. That will require a real effort, but we (BibLibre) agree to deal with it (and, reminder, it's already sponsored -thanks to a new french university switching to Koha- !) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
Hi, I want to ask specific question about this topic to Paul: Are you using Solr with Tomcat or Jetty as servlet container ? Bye -- Zeno Tajoli CILEA - Segrate (MI) tajoliAT_SPAM_no_prendiATcilea.it (Indirizzo mascherato anti-spam; sostituisci qaunto tra AT con @)
Le 11/10/2010 12:53, Zeno Tajoli a écrit :
Hi, I want to ask specific question about this topic to Paul: Are you using Solr with Tomcat or Jetty as servlet container ? This question is more for hdl than for me ;-)
I think it's tomcat, but let hdl confirm or give more details ! -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
Le 11/10/2010 16:29, Paul Poulain a écrit :
Le 11/10/2010 12:53, Zeno Tajoli a écrit :
Hi, I want to ask specific question about this topic to Paul: Are you using Solr with Tomcat or Jetty as servlet container ? This question is more for hdl than for me ;-)
I think it's tomcat, but let hdl confirm or give more details !
either tomcat or jetty could be suitable, depending on your requirements. Universities would rather use Tomcat. -- Henri-Damien LAURENT
One thing I'd like to see come out of the discussion is some idea of what we want from from a Search interface. One of the problems of finding shiny new solutions is that having chased after them you also import a lot of junk you then have to live with. (retrospectively we may have done that with zebra). I'm quite willing to believe that some of the perceived zebra problems are problems not with it per se but with how we handle it. When developing software it is a good method to write a test first that fails then write the functionality to pass that test. It means you have a clear target of what you are trying to achieve. We need to have a clear idea of what we want a search package to provide us and what constraints it must meet, otherwise we can't evaluate solutions (or even identify if the failure to meet these lies in the external software or in areas we can code our way out of them). Whatever we decide concerning zebra/Solr etc. I'd hope we come out with a clearer picture of what we need from them or from x. Colin -- Colin Campbell Chief Software Engineer, PTFS Europe Limited Content Management and Library Solutions +44 (0) 208 366 1295 (phone) +44 (0) 7759 633626 (mobile) colin.campbell@ptfs-europe.com skype: colin_campbell2 http://www.ptfs-europe.com
LAURENT Henri-Damien wrote:
involved in the community as previously. Paul promised some POCs, here is one available. [...]
Sorry for taking a while to look at this, but it raised so many questions in my mind when I first read it and I've been a bit busy, so I thought I'd leave it a while and see if some were covered by others. Some were (thanks!) but many are left, so here we go: What's a POC? Piece Of Code? (I assume it's not the C I'd usually mean in that abbreviation ;-) )
zebra is fast and embeds native z3950 server. But it has also some major drawbacks we have to cope with on our everyday life making it quite difficult to maintain.
1. zebra config files are a nightmare.
I've librarians editing zebra config files. They've seen far worse from the awful library management systems of the past.
You can't drive the configuration file easily. namely : Can't edit indexs via HTTP or configuration. all is in files hardcoded on disk.
We could fix this by providing an HTTP interface if anyone wanted. This isn't a problem unique to Zebra: some Koha configuration is only in files on disk. Being in a config file is not hardcoded! So, this is solvable if someone wanted it enough. Does anyone want me to take this enhancement forwards?
[...] And ppl donot get a translation of the indexes since all the indexes are hardcoded in the ccl.properties and we donot have a translation process so that ccl attributes could be translated into different languages.
This sounds like a problem in our translation process. Would the translation manager like to consider it, please?
2. no real-time indexing : the use of a crontab is poor: when you add an authority while creating a biblio, you have to wait some some minutes to end your biblio
This is being considered in bug 5165. It's a problem in how we are using Zebra, really.
(might be solved since zebra has some way to index biblios via z3950 extended services, but hard and should be tested and at the time community first tested that, a performance problem was raised on indexing.)
Does someone have a link to this performance problem, please?
3. no way to access/process/delete data easily. If you have indexes in it or have some problems with your data, you have to reindex the whole stuff and indexing errors are quite difficult to detect.
I'm not entirely sure what is being wanted here. Indexing problems have been a bit nasty on many systems.
4. during index process of a file, if you have a problem in your data, zebraidx just fails silently…
Example?
And this is NOT secure.
What security data does zebra leak in this failure case?
And you have no way to know WHICH biblio made the process crash. [...]
It's quite possible, but Koha has made that mistake too. In one recent less serious example, I found that Koha knew which biblio was at fault, but didn't bother to report the biblio details in the failure error message. However, if it's an actual crash, it can be difficult to generate an error from a crashing C process. Maybe you could dump core and examine it, but bissecting the input data like HDL did is probably quicker.
5. facets are not working properly : they are on the result displayed because there are problems with diacritics & facets that can't be solved as of today. And noone can provide a solution (we spoke about that with indexdata and no clear solution was really provided.
Does anyone have a link to that conversation, please? I'd like to know more about it before we hit it for real.
6. zebra does not evolve anymore. There is no real community around it, it's just an opensource indexdata software. We sent many questions onlist and never got answers. We could pay for better support but the fee required is quite deterrent and benefit is still questionable.
It's disappointing there's no community, but that happens sometimes. I guess we could try and make it part of our community, if it's important enough. It's some different skills, but not completely inconsistent. What fee is being asked for what benefit?
7. icu & zebra are colleagues, not really friends : right truncation not working, fuzzy search not working and facets.
Those are pretty specific claims, directly contradicting http://www.indexdata.com/zebra/doc/querymodel-rpn.html#querymodel-bib1-trunc... http://www.indexdata.com/zebra/doc/querymodel-zebra.html#querymodel-zebra-at... and so on. Anyone else like to comment on them?
8. we use a deprecated way to define indexes for biblios (grs1) and the tool developped by indexdata to change to DOM has many flaws. we could manage and do with it. But is it worth the strive ?
I think so.
I think that every one agrees that we have to refactor C4::Search. Indeed, query parser is not able to manage independantly all the configuration options. And usage of usmarc as internal for biblio comes with a serious limitation of 9999 bytes, which for big biblios with many items, is not enough.
Where does that 9999-byte limit come from? Could some methods from the analytic records RFC give us a route around it?
BibLibre investigated in a catalogue based on solr. A University in France contracted us for that development. This University is in relation with all the community here in France and solr will certainly be adopted by all the libraries France wide. [...]
That's disappointing. While it's not a problem for universities, the big problem I see with Solr http://lucene.apache.org/solr/ is that it is Java, which poses many management challenges for smaller libraries and requires very different skills to current Koha deployments. It seems a bit like throwing the baby out with the bath water, at first glance.
Solr indexes with data with HTTP.
Why is this the top benefit? We're smart enough to write for whatever protocol, so I'm not sure I understand.
It can provide fuzzy search, search on synonyms, suggestions It can provide facet search, stemming.
In theory, so can Zebra.
utf8 support is embedded.
Hmmm, we'll see. (I thought Java preferred some other unicode form.)
Community is really impressively reactive and numerous and efficient. And documentation is very good and exhaustive.
Well, those are both good things.
a) Librarians can define their own indexes, and there is a plugin that fetches data from rejected authorities and from authorised_values (that could/should have been achieved with zebra but only with major work on xslt).
I've read that to be so configurable online, Solr must be allowed to create files in its own installation directory, which seems like a security problem (or at least, against debian policy). Is that true? For example http://sourceforge.net/mailarchive/forum.php?thread_name=ABC31E122EEAC44897B337BDEA897736BE58B9663A%40VUEX2.vuad.villanova.edu&forum_name=vufind-general Can we predefine indexes? But it could be achieved with zebra?
b) C4/Search.pm count lines of code could be shrinked ten times. You can test from poc_solr branch on git://git.biblibre.com/koha_biblibre.git But you have to install solr.
In other questions, how does its performance compare? Are there drawbacks to adopting it, counterweighing the benefits of overcoming the above-stated problems with Zebra? Hope that helps move the discussion on, -- MJ Ray (slef), member of www.software.coop, a for-more-than-profit co-op. Past Koha Release Manager (2.0), LMS programmer, statistician, webmaster. In My Opinion Only: see http://mjr.towers.org.uk/email.html Available for hire for Koha work http://www.software.coop/products/koha
I think that every one agrees that we have to refactor C4::Search. Indeed, query parser is not able to manage independantly all the configuration options. And usage of usmarc as internal for biblio comes with a serious limitation of 9999 bytes, which for big biblios with many items, is not enough.
Where does that 9999-byte limit come from? Could some methods from the analytic records RFC give us a route around it?
The byte limit is a restriction of the ISO-2709 format. If we use MARCXML, we can avoid that, but as I understand it, Zebra currently just ingests binary MARC. Analytics support could help with this by moving item data off one biblio onto another, but there would be no guarantee of it working universally. Cheers, -Ian
Hi, 2010/11/11 Ian Walls <ian.walls@bywatersolutions.com>:
The byte limit is a restriction of the ISO-2709 format. If we use MARCXML, we can avoid that, but as I understand it, Zebra currently just ingests binary MARC.
This is incorrect. As configured by default for use with Koha, Zebra can ingest either ISO2709 blobs or MARCXML (e.g., by passing the -x option to rebuild_zebra.pl), so the ISO2709 record size limit is not a limitation of Zebra. Regards, Galen -- Galen Charlton gmcharlt@gmail.com
Ah, right, I'd forgotten about that switch, as it doesn't work with authorities, and I can't do rebuild_zebra.pl -a -b -x -z in my crontab. So, the ISO2709 character limit is not actually an issue at all, then. -Ian On Thu, Nov 11, 2010 at 9:10 AM, Galen Charlton <gmcharlt@gmail.com> wrote:
Hi,
2010/11/11 Ian Walls <ian.walls@bywatersolutions.com>:
The byte limit is a restriction of the ISO-2709 format. If we use MARCXML, we can avoid that, but as I understand it, Zebra currently just ingests binary MARC.
This is incorrect. As configured by default for use with Koha, Zebra can ingest either ISO2709 blobs or MARCXML (e.g., by passing the -x option to rebuild_zebra.pl), so the ISO2709 record size limit is not a limitation of Zebra.
Regards,
Galen -- Galen Charlton gmcharlt@gmail.com
-- Ian Walls Lead Development Specialist ByWater Solutions Phone # (888) 900-8944 http://bywatersolutions.com ian.walls@bywatersolutions.com Twitter: @sekjal
Le 11/11/2010 15:19, Ian Walls a écrit :
Ah, right, I'd forgotten about that switch, as it doesn't work with authorities, and I can't do rebuild_zebra.pl <http://rebuild_zebra.pl> -a -b -x -z in my crontab.
So, the ISO2709 character limit is not actually an issue at all, then.
Well, actually, it is in koha. Since Koha gets iso2709 from a zebra search. If it Koha would take marcxml, it would be even slower to get search results with the Search.pm as it stands now... And let me know if you know a librarian who would like that. -- Henri-Damien LAURENT
On 15 November 2010 06:25, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
Le 11/11/2010 15:19, Ian Walls a écrit :
Ah, right, I'd forgotten about that switch, as it doesn't work with authorities, and I can't do rebuild_zebra.pl <http://rebuild_zebra.pl> -a -b -x -z in my crontab.
So, the ISO2709 character limit is not actually an issue at all, then.
Well, actually, it is in koha. Since Koha gets iso2709 from a zebra search. If it Koha would take marcxml, it would be even slower to get search results with the Search.pm as it stands now... And let me know if you know a librarian who would like that.
Yep, there is utterly no doubt C4::Search needs a rewrite. This was a goal of 3.4. I'm pretty sure I mentioned it in my proposal, and we had some volunteers to work on it. I still think it is nessecary, but I do think it is better to do it in a way that allows for a structure like C4/Search.pm C4/Search/Nutch.pm C4/Search/Zebra.pm C4/Search/Solr.pm Or using searchengine or something else to achieve the same. Chris
Le 14/11/2010 18:32, Chris Cormack a écrit :
On 15 November 2010 06:25, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
Le 11/11/2010 15:19, Ian Walls a écrit :
Ah, right, I'd forgotten about that switch, as it doesn't work with authorities, and I can't do rebuild_zebra.pl <http://rebuild_zebra.pl> -a -b -x -z in my crontab.
So, the ISO2709 character limit is not actually an issue at all, then.
Well, actually, it is in koha. Since Koha gets iso2709 from a zebra search. If it Koha would take marcxml, it would be even slower to get search results with the Search.pm as it stands now... And let me know if you know a librarian who would like that.
Yep, there is utterly no doubt C4::Search needs a rewrite. This was a goal of 3.4. I'm pretty sure I mentioned it in my proposal, and we had some volunteers to work on it. Who ? What is their plan ? Where is the discussion about the rewrite ? Where is the code ? Is there a working group on that ? Is there any place for collaboration ? -- Henri-Damien LAURENT
On 15 November 2010 08:28, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
Le 14/11/2010 18:32, Chris Cormack a écrit :
On 15 November 2010 06:25, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
Le 11/11/2010 15:19, Ian Walls a écrit :
Ah, right, I'd forgotten about that switch, as it doesn't work with authorities, and I can't do rebuild_zebra.pl <http://rebuild_zebra.pl> -a -b -x -z in my crontab.
So, the ISO2709 character limit is not actually an issue at all, then.
Well, actually, it is in koha. Since Koha gets iso2709 from a zebra search. If it Koha would take marcxml, it would be even slower to get search results with the Search.pm as it stands now... And let me know if you know a librarian who would like that.
Yep, there is utterly no doubt C4::Search needs a rewrite. This was a goal of 3.4. I'm pretty sure I mentioned it in my proposal, and we had some volunteers to work on it. Who ? What is their plan ? Where is the discussion about the rewrite ? Where is the code ? Is there a working group on that ? Is there any place for collaboration ?
Wow, why so hostile, I just pointed out that it was in my RM proposal, it's on the wiki. The volunteering was done on irc, if i remember rightly and as far as I know no one has written a plan up yet. I don't see the need to jump on me for saying it. However, I still maintain that without a proper search engine abstraction layer, I would consider any work on Search to be unfinished. Chris
On 15 November 2010 08:34, Chris Cormack <chris@bigballofwax.co.nz> wrote:
On 15 November 2010 08:28, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
Le 14/11/2010 18:32, Chris Cormack a écrit :
On 15 November 2010 06:25, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
Le 11/11/2010 15:19, Ian Walls a écrit :
Ah, right, I'd forgotten about that switch, as it doesn't work with authorities, and I can't do rebuild_zebra.pl <http://rebuild_zebra.pl> -a -b -x -z in my crontab.
So, the ISO2709 character limit is not actually an issue at all, then.
Well, actually, it is in koha. Since Koha gets iso2709 from a zebra search. If it Koha would take marcxml, it would be even slower to get search results with the Search.pm as it stands now... And let me know if you know a librarian who would like that.
Yep, there is utterly no doubt C4::Search needs a rewrite. This was a goal of 3.4. I'm pretty sure I mentioned it in my proposal, and we had some volunteers to work on it. Who ? What is their plan ? Where is the discussion about the rewrite ? Where is the code ? Is there a working group on that ? Is there any place for collaboration ?
Wow, why so hostile, I just pointed out that it was in my RM proposal, it's on the wiki. The volunteering was done on irc, if i remember rightly and as far as I know no one has written a plan up yet. I don't see the need to jump on me for saying it.
However, I still maintain that without a proper search engine abstraction layer, I would consider any work on Search to be unfinished.
Ahh I realise I am to blame for some of the confusion. I haven't stated clearly, I do think adding Solr would be beneficial, and I would be happy to accept patches that do that, as long as they do it in a way that provides a search engine abstraction layer. I'd hate to see us lock ourselves hard to another search engine and have to redo it all again when we find a better one. If we are working towards DB abstraction, lets do Search engine abstraction too. Chris
Le 14/11/2010 20:34, Chris Cormack a écrit :
On 15 November 2010 08:28, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
Le 14/11/2010 18:32, Chris Cormack a écrit :
On 15 November 2010 06:25, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
Le 11/11/2010 15:19, Ian Walls a écrit :
Ah, right, I'd forgotten about that switch, as it doesn't work with authorities, and I can't do rebuild_zebra.pl <http://rebuild_zebra.pl> -a -b -x -z in my crontab.
So, the ISO2709 character limit is not actually an issue at all, then.
Well, actually, it is in koha. Since Koha gets iso2709 from a zebra search. If it Koha would take marcxml, it would be even slower to get search results with the Search.pm as it stands now... And let me know if you know a librarian who would like that.
Yep, there is utterly no doubt C4::Search needs a rewrite. This was a goal of 3.4. I'm pretty sure I mentioned it in my proposal, and we had some volunteers to work on it. Who ? What is their plan ? Where is the discussion about the rewrite ? Where is the code ? Is there a working group on that ? Is there any place for collaboration ?
Wow, why so hostile, I just pointed out that it was in my RM proposal, it's on the wiki. The volunteering was done on irc, if i remember rightly and as far as I know no one has written a plan up yet. I don't see the need to jump on me for saying it.
However, I still maintain that without a proper search engine abstraction layer, I would consider any work on Search to be unfinished.
It was simple questions, to get some information. It looked to me that you were saying that there was already a team working on C4::Search rewrite. I was not aware of that and saw no discussion on some ongoing work on C4::Search other than the ones on Solr, which we initiated. Friendly -- Henri-Damien LAURENT
Le 14/11/2010 18:32, Chris Cormack a écrit :
On 15 November 2010 06:25, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
Le 11/11/2010 15:19, Ian Walls a écrit :
Ah, right, I'd forgotten about that switch, as it doesn't work with authorities, and I can't do rebuild_zebra.pl <http://rebuild_zebra.pl> -a -b -x -z in my crontab.
So, the ISO2709 character limit is not actually an issue at all, then.
Well, actually, it is in koha. Since Koha gets iso2709 from a zebra search. If it Koha would take marcxml, it would be even slower to get search results with the Search.pm as it stands now... And let me know if you know a librarian who would like that.
Yep, there is utterly no doubt C4::Search needs a rewrite. This was a goal of 3.4. I'm pretty sure I mentioned it in my proposal, and we had some volunteers to work on it.
I still think it is nessecary, but I do think it is better to do it in a way that allows for a structure like C4/Search.pm C4/Search/Nutch.pm C4/Search/Zebra.pm C4/Search/Solr.pm
Or using searchengine or something else to achieve the same. Well. Our code is based on Data::SearchEngine, All we would have to do is writing a wrapper for Zebra, Nutch Whatever. And try and use that in the C4::Search. We took the burden to refactor the C4::Search. We wanted to test and make that work with Solr. If someone is volunteering for a Data::SearchEngine::Zebra, feel free...
We also just discovered something about zoom and zebra. It really has a problem when you are using persistent connexions to it. Be warned... Indeed, a) when doing more than 200 searches, searches are getting slower and slower. And memory footprint increases. b) there is no way to disconnect cleanly. One has to destroy the connection, (we patched C4::Context to ba able to destroy the connections) but destroy won't close cleanly the connection on the zebra server. and there will be a thread that will be left for each connection you open and close. -- Henri-Damien LAURENT
On 15 November 2010 10:14, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
Le 14/11/2010 18:32, Chris Cormack a écrit :
On 15 November 2010 06:25, LAURENT Henri-Damien <henridamien.laurent@biblibre.com> wrote:
Le 11/11/2010 15:19, Ian Walls a écrit :
Ah, right, I'd forgotten about that switch, as it doesn't work with authorities, and I can't do rebuild_zebra.pl <http://rebuild_zebra.pl> -a -b -x -z in my crontab.
So, the ISO2709 character limit is not actually an issue at all, then.
Well, actually, it is in koha. Since Koha gets iso2709 from a zebra search. If it Koha would take marcxml, it would be even slower to get search results with the Search.pm as it stands now... And let me know if you know a librarian who would like that.
Yep, there is utterly no doubt C4::Search needs a rewrite. This was a goal of 3.4. I'm pretty sure I mentioned it in my proposal, and we had some volunteers to work on it.
I still think it is nessecary, but I do think it is better to do it in a way that allows for a structure like C4/Search.pm C4/Search/Nutch.pm C4/Search/Zebra.pm C4/Search/Solr.pm
Or using searchengine or something else to achieve the same. Well. Our code is based on Data::SearchEngine, All we would have to do is writing a wrapper for Zebra, Nutch Whatever. And try and use that in the C4::Search. We took the burden to refactor the C4::Search. We wanted to test and make that work with Solr. If someone is volunteering for a Data::SearchEngine::Zebra, feel free...
Yes, please do (someone). Because this will make it much much more likely the Solr work is accepted into master for 3.4 Chris
So, the ISO2709 character limit is not actually an issue at all, then. Well, actually, it is in koha. Since Koha gets iso2709 from a zebra search. If it Koha would take marcxml, it would be even slower to get search results with the Search.pm as it stands now... And let me know if you know a librarian who would like that.
MARCXML parsing is slow because MARC::File::XML uses a SAX parser to do the job and do some 'magic' encoding-decoding to-from MARC8--Galen could correct me if I'm wrong. But since records stored into Koha are cleanly UTF-8 encoded, are well formed XML and respect a minimalist schema, we could parse them much more quickly directly in Perl. I've done some experimentation. It works easily. This code could be ported in five minutes: http://tinyurl.com/3x3d6b9 -- Frédéric
Hi, 2010/11/14 Frédéric Demians <frederic@tamil.fr>:
MARCXML parsing is slow because MARC::File::XML uses a SAX parser to do the job and do some 'magic' encoding-decoding to-from MARC8--Galen could correct me if I'm wrong.
Properly used (i.e., with BinaryEncoding => utf8 when parsing known UTF-8 MARCXML records), MARC::File::XML doesn't automatically transcode from MARC-8 to UTF-8, so that's a nonissue. Admittedly, there are still some circumstances where MARC::File::XML does inappropriately try to inject a MARC-8 to UTF-8 conversion. Patches to improve MARC::File::XML are welcome.
But since records stored into Koha are cleanly UTF-8 encoded, are well formed XML and respect a minimalist schema,
That is the ideal. In practice, Koha currently does not enforce either of your two assumptions in that statement; patches to tighten that up would be a good idea.
we could parse them much more quickly directly in Perl.
I question whether a pure Perl implementation would be faster. LibXML::XML::SAX, XML::SAX::Expat, and XML::SAX::ExpatXS have the the advantage that much of the parsing is handled by C code.
I've done some experimentation. It works easily. This code could be ported in five minutes:
Are you suggesting that we adopt yet another hand-crafted, pure Perl XML parser, one that does not support namespaces (a lot of MARCXML data in the wild does reference the marc namespace) and does not check for well formed XML *and* adopt a new MARC module that appears to be all of a few days old and lacks test cases for use in Koha? What you propose is interesting, and I'm sure you'll pursue it, but it would need more time to bake. On a more general note, XML parsing is a (mostly) solved problem in Perl. I don't think the way forward is to interpose hand-crafted pure-Perl-parsing of the MARCXML. To suggest an alternative approach that I think would would bear fruit and be less error prone, we can try other standard XML parsers such as XML::Twig. Regards, Galen -- Galen Charlton gmcharlt@gmail.com
Le 14/11/2010 22:28, Galen Charlton a écrit :
Hi,
2010/11/14 Frédéric Demians <frederic@tamil.fr>:
MARCXML parsing is slow because MARC::File::XML uses a SAX parser to do the job and do some 'magic' encoding-decoding to-from MARC8--Galen could correct me if I'm wrong.
Properly used (i.e., with BinaryEncoding => utf8 when parsing known UTF-8 MARCXML records), MARC::File::XML doesn't automatically transcode from MARC-8 to UTF-8, so that's a nonissue. Admittedly, there are still some circumstances where MARC::File::XML does inappropriately try to inject a MARC-8 to UTF-8 conversion. Patches to improve MARC::File::XML are welcome.
But since records stored into Koha are cleanly UTF-8 encoded, are well formed XML and respect a minimalist schema,
That is the ideal. In practice, Koha currently does not enforce either of your two assumptions in that statement; patches to tighten that up would be a good idea. Some work on it is pushed in BibLibre-various branch. C4::Charset, C4::Biblio and C4::Search. We used that to get Korean correctly displayed... and searched.
we could parse them much more quickly directly in Perl.
On a more general note, XML parsing is a (mostly) solved problem in Perl. I don't think the way forward is to interpose hand-crafted pure-Perl-parsing of the MARCXML. To suggest an alternative approach that I think would would bear fruit and be less error prone, we can try other standard XML parsers such as XML::Twig.
Having played with that module, it is quite neat and fast at parsing, was meant for parsing on the fly big XML documents. But again, we would have to improve C4::XSLT.pm... -- Henri-Damien LAURENT
But since records stored into Koha are cleanly UTF-8 encoded, are well formed XML and respect a minimalist schema, That is the ideal. In practice, Koha currently does not enforce either of your two assumptions in that statement; patches to tighten that up would be a good idea.
I don't understand. Do you mean that biblioitems.marcxml field and its mirror in Zebra can contain something else than valid MARCXML? Invalid encoded characters shouldn't change anything whatever parser is used. I see bug #2916 on bugzilla. Is there something more?
we could parse them much more quickly directly in Perl. I question whether a pure Perl implementation would be faster. LibXML::XML::SAX, XML::SAX::Expat, and XML::SAX::ExpatXS have the the advantage that much of the parsing is handled by C code.
My tests show that pure Perl parsing is twelve time as fast as with a SAX parser. A script test is here: http://tinyurl.com/23xaqkg
Are you suggesting that we adopt yet another hand-crafted, pure Perl XML parser, one that does not support namespaces (a lot of MARCXML data in the wild does reference the marc namespace) and does not check for well formed XML *and* adopt a new MARC module that appears to be all of a few days old and lacks test cases for use in Koha?
I've neither said nor suggested that. The issue pointed by Henri-Damien is that in C4::Search we get MARC::Record from ISO2709 because using MARCXML to build the same MARC::Record is much slower. And so we're limited to 99,999 record size. I say that we could build a MARC::Record from the MARCXML returned by Zebra using a pure Perl parser. And so I pointed to some code explaining that it could be ported, ie adapted to generate a MARC::Record as usually used in Koha. Have I proposed to substitute a new (immature?) MARC module, for whatever motives? I don't think so.
On a more general note, XML parsing is a (mostly) solved problem in Perl. I don't think the way forward is to interpose hand-crafted pure-Perl-parsing of the MARCXML. To suggest an alternative approach that I think would would bear fruit and be less error prone, we can try other standard XML parsers such as XML::Twig.
A MARCXML document is very simple XML which doesn't need a full fledged XML parser. I'm just saying that as soon as MARCXML records as stored in Koha are valid, if it isn't already the case, we can avoid using an heavy-weighted parser which impact performance and isn't required. We need of course to continue to use a SAX parser for incoming records. -- Frédéric
Hi, 2010/11/14 Frédéric Demians <frederic@tamil.fr>:
generate a MARC::Record as usually used in Koha. Have I proposed to substitute a new (immature?) MARC module, for whatever motives? I don't think so.
Your example code is a Marc::Moose module, no? Regards, Galen -- Galen Charlton gmcharlt@gmail.com
generate a MARC::Record as usually used in Koha. Have I proposed to substitute a new (immature?) MARC module, for whatever motives? I don't think so.
Your example code is a Marc::Moose module, no?
Yes, that's why I've said since my first email that it could be ported to be used with MARC::Record. And this is finally what you have done in your tests. There was no call at all for changing the MARC module used in Koha. Misunderstandings may arise in an international communitarian project like Koha from the fact that all participants are not native English speakers, which is my case. Thanks. -- Frédéric
Hi, 2010/11/14 Frédéric Demians <frederic@tamil.fr>:
A MARCXML document is very simple XML which doesn't need a full fledged XML parser. I'm just saying that as soon as MARCXML records as stored in Koha are valid, if it isn't already the case, we can avoid using an heavy-weighted parser which impact performance and isn't required. We need of course to continue to use a SAX parser for incoming records.
I've measured, and your parser is, in fact pretty fast -- *if* you feed it only MARCXML that meets narrower constraints than are permitted by the MARC21slim schema. However, I see no good reason to limit Koha to that artificial restriction; having biblioitems.marcxml contain MARCXML that validates against the MARC21slim is sufficient. Two parsers doing similar operations is an invitation for subtle bugs. The pure Perl parser you propose currently doesn't handle namespaces prefixes (which are allowed in MARC21slim records), wouldn't handle any situation where the attributes aren't in the order you expect them in (attribute order is not significant per the XML specification), and will blithely accept non-well-formed XML without complaining (this is *not* a good thing). It also doesn't recognize and correctly handle XML entities. Obviously you could address much of this in your code, but I suspect what you'll find is that you'll end up with an XML parser that is slower and still has more bugs than any of the standard parser modules. Fortunately, I've found an approach that is significantly faster than MARC::File::XML/SAX: dropping SAX from MARC::File::XML entirely and using XML::LibXML's DOM parser instead [1]. It is faster [2] than using XML::LibXML::SAX::Parser [3], XML::Expat [4], and even XML::ExpatXS [5]. A pure Perl approach based on your work [6] does win the race [7], but it also fails some of MARC::File::XML's test cases and I'm sure it would lose speed once extended to handle the full range of what constitutes a valid MARCXML document. But, one might ask, what about memory usage with a DOM parser? MARC::File::XML as used by Koha (and used in general) is geared towards parsing one record at a time; it doesn't currently have any provision for loading an entire file in memory. A DOM tree for a typical MARCXML record is not a big deal, and even a record having several thousand items wouldn't be any more unmanageable. (Of course, as we all know, one of the most significant gains to be had will arise from changing Koha so that it doesn't embed item data in bib MARC tags as a matter of course). In fact, we already have proof that we'd be no worse off as far as memory consumption is concerned -- XML::LibXML::SAX::Parser, as it happens, isn't a traditional SAX parser. What it does is load the XML document into a DOM tree, then walks the tree and fires off SAX events. In other words, we're *already* using a DOM parser. In any event, I would be grateful for people to test the DOM version of MARC::File::XML. It passes MARC::File::XML's test suite successfully, but more testing to verify that it won't break things would help a great deal. By the way, I did also try XML::Twig, but that didn't turn out to be faster than XML::LibXML::SAX::Parser, and in some cases was slower. [1] http://git.librarypolice.com/?p=marcpm.git;a=shortlog;h=refs/heads/use-dom-i... [2] http://librarypolice.com/nytprof/run-libxml-dom-2/index.html [3] http://librarypolice.com/nytprof/run-sax-libxml-sax-parser/ [4] http://librarypolice.com/nytprof/run-sax-expat/index.html [5] http://librarypolice.com/nytprof/run-sax-expatxs/index.html [6] http://git.librarypolice.com/?p=marcpm.git;a=shortlog;h=refs/heads/pure-perl [7] http://librarypolice.com/nytprof/run-pp/ Regards, Galen -- Galen Charlton gmcharlt@gmail.com
Hi 2010/11/14 Galen Charlton <gmcharlt@gmail.com>:
[1] http://git.librarypolice.com/?p=marcpm.git;a=shortlog;h=refs/heads/use-dom-i... [2] http://librarypolice.com/nytprof/run-libxml-dom-2/index.html [3] http://librarypolice.com/nytprof/run-sax-libxml-sax-parser/ [4] http://librarypolice.com/nytprof/run-sax-expat/index.html [5] http://librarypolice.com/nytprof/run-sax-expatxs/index.html [6] http://git.librarypolice.com/?p=marcpm.git;a=shortlog;h=refs/heads/pure-perl [7] http://librarypolice.com/nytprof/run-pp/
By the way, the profiles are coming from runs of the following script applied to a MARCXML file containing 5,000 records, with various combinations of SAX parsers and the DOM and PP code for MARC::File::XML. #!/usr/bin/perl use MARC::File::XML (BinaryEncoding => 'utf8'); use MARC::Record; use MARC::Batch; binmode STDOUT, ':utf8'; my $batch = MARC::Batch->new('XML', $ARGV[0]); while (my $record = $batch->next) { $i++; print $record->as_usmarc(); if ($i % 1000 == 0) { print STDERR "$i ", scalar(localtime), "\n"; } last if $i == 5000; } Regards, Galen -- Galen Charlton gmcharlt@gmail.com
And for those who want to run test by their self, here attached is my tests comparing pure Perl parsing and various SAX parser (which need to be installed): XML::SAX::PurePerl XML::LibXML::SAX::Parser XML::SAX::Expat XML::SAX::ExpatXS SAX parsing is done directly, without using MARC::File::XML in order to have raw figures. Parsing in MARC::File::XML should slow down a little bit but I can't say of what magnitude.
2010/11/15 Frédéric Demians <frederic@tamil.fr>:
And for those who want to run test by their self, here attached is my tests comparing pure Perl parsing and various SAX parser (which need to be installed):
XML::SAX::PurePerl XML::LibXML::SAX::Parser XML::SAX::Expat XML::SAX::ExpatXS
SAX parsing is done directly, without using MARC::File::XML in order to have raw figures. Parsing in MARC::File::XML should slow down a little bit but I can't say of what magnitude.
You could add the test using DOM also IE just XML::LibXML (Without the sax) which we now know is a lot faster :) Galen's tests essentially do the same thing, except with passing it through MARC::File::XML Chris Chris
IE just XML::LibXML (Without the sax) which we now know is a lot faster :) Galen's tests essentially do the same thing, except with passing it through MARC::File::XML
Comparisons are odious. DOM uses an underlying SAX parser to load any XML document in memory. DOM is not as SAX parser as itself. Galen tests, as I understand them, show that current MARC::File::XML parser, which include a specif SAX event handler, is slower than loading directly a DOM document. It contradicts the theory. The explanation is, as stated by Galen, that Perl SAX parser implementation is not good... My tests use XML::Simple and so load the whole MARCXML document in memory before rendering it into a MARC::Record object. It gives a picture of the difference between parsing MARCXML in pure Perl vs using an external SAX parser. -- Frédéric
2010/11/15 Frédéric Demians <frederic@tamil.fr>:
IE just XML::LibXML (Without the sax) which we now know is a lot faster :) Galen's tests essentially do the same thing, except with passing it through MARC::File::XML
Comparisons are odious. DOM uses an underlying SAX parser to load any XML document in memory. DOM is not as SAX parser as itself. Galen tests, as I understand them, show that current MARC::File::XML parser, which include a specif SAX event handler, is slower than loading directly a DOM document. It contradicts the theory. The explanation is, as stated by Galen, that Perl SAX parser implementation is not good... My tests use XML::Simple and so load the whole MARCXML document in memory before rendering it into a MARC::Record object. It gives a picture of the difference between parsing MARCXML in pure Perl vs using an external SAX parser.
I work with the author of XML::Simple .. and he would (and does) tell people not to use it for anything than parsing very simple XML structures. http://search.cpan.org/~grantm/XML-Simple-2.18/lib/XML/Simple.pm#WHERE_TO_FR... So do I understand from what you are saying, that Galens work is not useful, and that a pureperl XML parser is the only way forward? I hope this is just another language based misunderstanding. Because I disagree totally if not. Chris
I work with the author of XML::Simple .. and he would (and does) tell people not to use it for anything than parsing very simple XML structures.
Which is the case when benchmarking a simple MARCXML document parsing switching quickly from one underlying SAX parser to the other.
So do I understand from what you are saying, that Galens work is not useful, and that a pureperl XML parser is the only way forward?
No. I'm not saying Galen's work is not useful.
I hope this is just another language based misunderstanding. Because I disagree totally if not.
Being the English native speaker, you have to do the effort to not misunderstand as I have to do my best to be understood. Thanks. -- Frédéric
2010/11/15 Frédéric Demians <frederic@tamil.fr>:
I work with the author of XML::Simple .. and he would (and does) tell people not to use it for anything than parsing very simple XML structures.
Which is the case when benchmarking a simple MARCXML document parsing switching quickly from one underlying SAX parser to the other.
So do I understand from what you are saying, that Galens work is not useful, and that a pureperl XML parser is the only way forward?
No. I'm not saying Galen's work is not useful.
I hope this is just another language based misunderstanding. Because I disagree totally if not.
Being the English native speaker, you have to do the effort to not misunderstand as I have to do my best to be understood.
That is what I am attempting to do and for the record, I spoke Maori before I spoke English. Another misunderstanding :) Chris
misunderstand as I have to do my best to be understood.
That is what I am attempting to do and for the record, I spoke Maori before I spoke English. Another misunderstanding :)
So thanks for trying to understand. And be ye merciful, even as your Maori (metaphoric) Father is merciful.
Just to throw in on something I ready earlier in this thread, I'd say that for a general practice with Koha going forward, we should pick a single XML parser that can handle arbitrary schemas, and use that. I would very much like to make Koha not just MARC-agnostic, but metadata schema agnostic, and coding ourselves into a corner now (even for a noticeable performance boost), would make life difficult later. As I think the rest of the thread attests, there are other ways to improve our XML parsing. If this had already been resolved earlier in the conversation, I apologize for redundancy; I haven't had my morning coffee yet. -Ian -- Ian Walls Lead Development Specialist ByWater Solutions Phone # (888) 900-8944 http://bywatersolutions.com ian.walls@bywatersolutions.com Twitter: @sekjal
[Much of the discussion of record parsers has very little to do with the subject of Solr/Lucene specifically under which preceding discussion of record parsers appeared.] Reply inline: Previous Subject: Re: [Koha-devel] Search Engine Changes : let's get some solr 1. GENERAL PUROSE XML PARSER. On Mon, November 15, 2010 13:58, Ian Walls wrote:
Just to throw in on something I ready earlier in this thread, I'd say that for a general practice with Koha going forward, we should pick a single XML parser that can handle arbitrary schemas, and use that.
Having a general purpose XML parser would be very useful as one step towards greater generalisation and abstraction in Koha. Picking a single XML parser for all use cases might be an optimisation mistake which we would come to regret in future. 2. METADATA SCHEMA AGNOSTIC RECORDS.
I would very much like to make Koha not just MARC-agnostic, but metadata schema agnostic, and coding ourselves into a corner now (even for a noticeable performance boost), would make life difficult later. As I think the rest of the thread attests, there are other ways to improve our XML parsing.
If this had already been resolved earlier in the conversation, I apologize for redundancy; I haven't had my morning coffee yet.
The issue of a general purpose XML parser had been considered tangentially but without the appropriate context of metadata schema agnostic records. I think that considering record parsers which are not MARC or MARCXML specific is important for long term development. 2.1. INTERNAL RECORD FORMATS. For some future development, Koha should not be dependent upon a metadata exchange record syntax for anything other than lossless data input and data output. An internal record syntax should be optimised for particular library management system functions. The general state of Koha may not be ready for the work which would be required to ensure that changing the base record format would be lossless. However, we should be enabling the future possibility by implementing abstraction when opportunities arise. Frédéric Demians recognises the distinction between internal record use for Koha and external record use for interfacing with the world. Previous discussion in the "MARC record size limit thread" had also considered non-XML record syntaxes such as YAML. On Mon, November 15, 2010 05:56, Frédéric Demians wrote: [...]
It's a design choice. MARCXML is the Koha internal serialization format for MARC records. There is no obligation to conform to MARC21slim schema. We even could choose another serialization format as it has already been discussed. biblioitems.marcxml isn't open to the wide.
[...]
And we could benefit of it if pure Perl parsing is a real performance gain. That is for the good reason.
However, the prospect of using Koha specific record syntax parser for record creation or modification scares me. I would much prefer some lower efficiency with validity constraints from a Perl module widely tested outside of Koha. 2.1.1. REASON FOR INTERNAL RECORD FORMATS. An example record format is record format optimised for indexing which would store information such as the language of material a clear appropriate place for indexing. Records optimised for indexing would be different from the primary form of the record optimised editing and an alternate form optimised for display. MARC often uses one or more of several different places with varying forms of presentation for the same information. Examples include language of material which may be multiple and refer to language from which material was translated; the muddle of recording content type, material type, carrier type and their various relationships; the muddle of date forms and similar numeric and sequential designators; the muddle of ordered classification and similar hierarchical designators; transcribed and natural language record content with no controlled vocabulary; etc. [In the interest of time, I omit providing detailed examples.] Consider the case of language of material. Enhancing records to use fixed fields or fixed subfields for better indexing is insufficient to record the complexity of language use cases. XPath indexing of MARC records cannot cope well enough with all the possibilities. The information can be parsed out of MARC records reliably into a record specially optimised for indexing. Storing the information in MARC in an easily indexable manner is the problem. 3. ENABLING FUTURE DEVELOPMENT. Generalising and abstracting record parsing would enable future development such as records normalised for a particular purpose without being dependent upon MARC. Developments which enable future work do not require a commitment to a particular development idea but help free the constraints of development practicalities by leaving less work to provide some future development. Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783
Thanks a lot for those thorough tests. Your optimization of MARCXML records parsing looks fantastic.
I've measured, and your parser is, in fact pretty fast -- *if* you feed it only MARCXML that meets narrower constraints than are permitted by the MARC21slim schema. However, I see no good reason to limit Koha to that artificial restriction; having biblioitems.marcxml contain MARCXML that validates against the MARC21slim is sufficient.
It's a design choice. MARCXML is the Koha internal serialization format for MARC records. There is no obligation to conform to MARC21slim schema. We even could choose another serialization format as it has already been discussed. biblioitems.marcxml isn't open to the wide. It is written by C4::ModBiblioMarc which uses MARC::Record::as_xml_record function to populate marcxml DB field. So we already have an internal restricted version of MARC21slim schema. And we could benefit of it if pure Perl parsing is a real performance gain. That is for the good reason.
Two parsers doing similar operations is an invitation for subtle bugs. The pure Perl parser you propose currently doesn't handle namespaces prefixes (which are allowed in MARC21slim records), wouldn't handle any situation where the attributes aren't in the order you expect them in (attribute order is not significant per the XML specification), and will blithely accept non-well-formed XML without complaining (this is *not* a good thing). It also doesn't recognize and correctly handle XML entities. Obviously you could address much of this in your code, but I suspect what you'll find is that you'll end up with an XML parser that is slower and still has more bugs than any of the standard parser modules.
See above. I don't see the need to handle any MARC21slim peculiarity in the limited needs of Koha internal functions. Regards, -- Frédéric
* Frédéric Demians (frederic@tamil.fr) wrote:
Thanks a lot for those thorough tests. Your optimization of MARCXML records parsing looks fantastic.
They sure do, I'll be testing out your git branch you have pushed for MARC/Perl http://marcpm.git.sourceforge.net/git/gitweb.cgi?p=marcpm/marcpm;a=shortlog;...
I've measured, and your parser is, in fact pretty fast -- *if* you feed it only MARCXML that meets narrower constraints than are permitted by the MARC21slim schema. However, I see no good reason to limit Koha to that artificial restriction; having biblioitems.marcxml contain MARCXML that validates against the MARC21slim is sufficient.
It's a design choice. MARCXML is the Koha internal serialization format for MARC records. There is no obligation to conform to MARC21slim schema. We even could choose another serialization format as it has already been discussed. biblioitems.marcxml isn't open to the wide. It is written by C4::ModBiblioMarc which uses MARC::Record::as_xml_record function to populate marcxml DB field. So we already have an internal restricted version of MARC21slim schema. And we could benefit of it if pure Perl parsing is a real performance gain. That is for the good reason.
I think that getting speed and compliance to the standard is the best of both worlds. If we store standard compliant MARCXML then our export routine is trivially easy :) Not to mention the benefit of being able to say we store MARCXML compliant to the standard. Chris -- Chris Cormack Catalyst IT Ltd. +64 4 803 2238 PO Box 11-053, Manners St, Wellington 6142, New Zealand
I think that getting speed and compliance to the standard is the best of both worlds. If we store standard compliant MARCXML then our export routine is trivially easy :) Not to mention the benefit of being able to say we store MARCXML compliant to the standard.
Koha MARCXML being a subset of MARC21slim is de facto compliant.
* Frédéric Demians (frederic@tamil.fr) wrote:
I think that getting speed and compliance to the standard is the best of both worlds. If we store standard compliant MARCXML then our export routine is trivially easy :) Not to mention the benefit of being able to say we store MARCXML compliant to the standard.
Koha MARCXML being a subset of MARC21slim is de facto compliant.
That's as may be, but using a MARC::Record that deals with fully MARC21slim compatible records in a fast manner is surely a good thing. Anyway lets not let it distract us from testing the DOM implementation which looks like a very promising development and a good path forward for MARCPM, hopefully lots of people take up the invitation to test. We should be able to use it to help us when we are ingesting records (either via Z3950 or bulkimport) to check for compliance, and in a fast manner :) Chris -- Chris Cormack Catalyst IT Ltd. +64 4 803 2238 PO Box 11-053, Manners St, Wellington 6142, New Zealand
Frédéric Demians wrote (quoting someone without giving credit):
But since records stored into Koha are cleanly UTF-8 encoded, are well formed XML and respect a minimalist schema, That is the ideal. In practice, Koha currently does not enforce either of your two assumptions in that statement; patches to tighten that up would be a good idea.
I don't understand. Do you mean that biblioitems.marcxml field and its mirror in Zebra can contain something else than valid MARCXML? Invalid encoded characters shouldn't change anything whatever parser is used. I see bug #2916 on bugzilla. Is there something more?
Well, 2916 is a description of the general problem, but there appear to be multiple vectors for this invalid MARCXML to get in there. If I remember correctly, I don't think it reaches the Zebra mirror because what goes there is MARC that is generated from the MARCXML, so no valid marcxml field means no MARC for Zebra. Hope that helps, -- MJ Ray (slef), member of www.software.coop, a for-more-than-profit co-op. Past Koha Release Manager (2.0), LMS programmer, statistician, webmaster. In My Opinion Only: see http://mjr.towers.org.uk/email.html Available for hire for Koha work http://www.software.coop/products/koha
MJ Ray schreef op do 11-11-2010 om 13:09 [+0000]:
4. during index process of a file, if you have a problem in your data, zebraidx just fails silently… Example?
When Zebra has a record that it chokes on, it will sometimes segfault. This is a problem in itself, a second part of that problem (and this is one we can fix) is that rebuild_zebra doesn't notice this. It should really start screaming about it. When zebra fails (segfaulting, or just not liking some data) it does things like refuse to process anything else from that point on, and it can be quite a time consuming process to track down exactly what record it is that's causing the issue.
And this is NOT secure. What security data does zebra leak in this failure case?
There's more than one definition of the word security. It's not secure in the same sense that a wheel on a car might not be secure. When it comes loose and goes flying, things don't work so well. (Although in the case of Zebra, you may not notice for a while.) -- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5957 6D23 8B16 EFAB FEF8 7175 14D3 6485 A99C EB6D
Le 11/11/2010 22:58, Robin Sheat a écrit :
MJ Ray schreef op do 11-11-2010 om 13:09 [+0000]:
4. during index process of a file, if you have a problem in your data, zebraidx just fails silently… Example?
When Zebra has a record that it chokes on, it will sometimes segfault. This is a problem in itself, a second part of that problem (and this is one we can fix) is that rebuild_zebra doesn't notice this. It should really start screaming about it.
When zebra fails (segfaulting, or just not liking some data) it does things like refuse to process anything else from that point on, and it can be quite a time consuming process to track down exactly what record it is that's causing the issue.
And this is NOT secure. What security data does zebra leak in this failure case?
There's more than one definition of the word security. It's not secure in the same sense that a wheel on a car might not be secure. When it comes loose and goes flying, things don't work so well. (Although in the case of Zebra, you may not notice for a while.)
Well, librarians consider the search engine to be more compared to the reactor of a plane rather than the wheel of a car. But I hope that MJ got the idea. In fact, the only way to get some information from zebrasrv is to catch the warn " previous transaction doesn't reach commit" in the logs from zebrasrv... And if you are indexing 10000 records, and you have one record which cause that error, your whole bunch of records is not indexed for want of maybe only one record (maybe more maybe less, who knows ? zebra stops at the very first without telling you which it is and without indexing the part that worked.).... So sometimes, in zebraqueue, things are marked as indexed... while they are not.... You may think it is ok for a good while and realize it is not. You are on a plane up to 10000 feet in the sky, you never know when your engine will blow out or stop. No warnings (from the crontab), no way to know that it is working. If it stops, hold on to your stick. -- Henri-Damien LAURENT
Reply inline: On Thu, November 11, 2010 13:09, MJ Ray wrote:
LAURENT Henri-Damien wrote:
involved in the community as previously. Paul promised some POCs, here is one available. [...]
Sorry for taking a while to look at this, but it raised so many questions in my mind when I first read it and I've been a bit busy, so I thought I'd leave it a while and see if some were covered by others. Some were (thanks!) but many are left, so here we go: What's a POC? Piece Of Code? (I assume it's not the C I'd usually mean in that abbreviation ;-) )
I do not remember seeing this question answered. The abbreviation POC had confused me when I first saw it. I think that the recent use of POC in our context is for 'proof of concept'. Use of that abbreviation is uncommon to my knowledge in English even in the context of software development. However, my experience may be lacking. I personally try to minimise my use of abbreviations where I think that some expected readers may not know them. I try to give them as a parenthetical to the full term in first use when I need an abbreviation to avoid excessive repetition. I understand that most people would not take the time to be as careful as I try to be when writing. Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783
I am less than thrilled with many aspects of Zebra. It would suit my purposes better if it had more graceful failure modes, parallelization, simpler configuration, etc. However, I am extremely wary of any replacement which involves Java. For what it does, zebrasrv runs very lean in terms of RAM and CPU utilization. For a single install the extra overhead of using a Java-based search engine may not be too expensive to compensate for. With mass-hosting that is not going to being the case, though, as all that overhead quickly adds up, and that's the use case I'm most concerned with. A possible workaround for this problem would be if multiple Koha instances' records could be stored and queried discretely from within a single SolR process. Do you know how difficult that would be to implement? I am in favor of modularizing record search capabilities in any case. Clay On Mon, Oct 4, 2010 at 1:10 AM, LAURENT Henri-Damien < henridamien.laurent@biblibre.com> wrote:
Hi As you already read in Paul previous message about "BibLibre strategy for 3.4 and next version", we are growing, want be involved in the community as previously. Paul promised some POCs, here is one available. We also worked on Plack and support. We created a base of script to search for Memoryleaks. We'll demonstrate that later.
zebra is fast and embeds native z3950 server. But it has also some major drawbacks we have to cope with on our everyday life making it quite difficult to maintain.
1. zebra config files are a nightmare. You can't drive the configuration file easily. namely : Can't edit indexs via HTTP or configuration. all is in files hardcoded on disk. => you can't list indexes you can't change indexes, you can't edit indexes, you can't say I want this index at OPAC, that in intranet. (Could be done with scraping ccl.properties, and then record.abs and bib1.att.... But what a HELL) So you cannot customize configuration defining the indexes you want easily. And ppl donot get a translation of the indexes since all the indexes are hardcoded in the ccl.properties and we donot have a translation process so that ccl attributes could be translated into different languages.
2. no real-time indexing : the use of a crontab is poor: when you add an authority while creating a biblio, you have to wait some some minutes to end your biblio (might be solved since zebra has some way to index biblios via z3950 extended services, but hard and should be tested and at the time community first tested that, a performance problem was raised on indexing.)
3. no way to access/process/delete data easily. If you have indexes in it or have some problems with your data, you have to reindex the whole stuff and indexing errors are quite difficult to detect.
4. during index process of a file, if you have a problem in your data, zebraidx just fails silently... And this is NOT secure. And you have no way to know WHICH biblio made the process crash. We had a LOT of trouble with Aix-Marseille universities that have some arabic translitterated biblios that makes zebra/icu completly crash ! We had to do some recursive script to find 14 biblios on 730 000 that makes zebra crash (even is properly stored & displayed)
5. facets are not working properly : they are on the result displayed because there are problems with diacritics & facets that can't be solved as of today. And noone can provide a solution (we spoke about that with indexdata and no clear solution was really provided.
6. zebra does not evolve anymore. There is no real community around it, it's just an opensource indexdata software. We sent many questions onlist and never got answers. We could pay for better support but the fee required is quite deterrent and benefit is still questionable.
7. icu & zebra are colleagues, not really friends : right truncation not working, fuzzy search not working and facets.
8. we use a deprecated way to define indexes for biblios (grs1) and the tool developped by indexdata to change to DOM has many flaws. we could manage and do with it. But is it worth the strive ?
I think that every one agrees that we have to refactor C4::Search. Indeed, query parser is not able to manage independantly all the configuration options. And usage of usmarc as internal for biblio comes with a serious limitation of 9999 bytes, which for big biblios with many items, is not enough.
BibLibre investigated in a catalogue based on solr. A University in France contracted us for that development. This University is in relation with all the community here in France and solr will certainly be adopted by all the libraries France wide. We are planning to release the code on our git early spring next year and rebase on whatever Koha version will be released at that time 3.4 or 3.6.
Why ?
Solr indexes with data with HTTP. It can provide fuzzy search, search on synonyms, suggestions It can provide facet search, stemming. utf8 support is embedded. Community is really impressively reactive and numerous and efficient. And documentation is very good and exhaustive.
You can see the results on solr.biblibre.com and catalogue.solr.biblibre.com
http://catalogue.solr.biblibre.com/cgi-bin/koha/opac-search.pl?q=jean http://solr.biblibre.com/cgi-bin/koha/admin/admin-home.pl you can log there with demo/demo lgoin/password
http://solr.biblibre.com/cgi-bin/koha/solr/indexes.pl is the page where ppl can manage their indexes and links.
a) Librarians can define their own indexes, and there is a plugin that fetches data from rejected authorities and from authorised_values (that could/should have been achieved with zebra but only with major work on xslt).
b) C4/Search.pm count lines of code could be shrinked ten times. You can test from poc_solr branch on git://git.biblibre.com/koha_biblibre.git But you have to install solr.
Any feedback/idea welcome. -- Henri-Damien LAURENT BibLibre _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
A possible workaround for this problem would be if multiple Koha instances' records could be stored and queried discretely from within a single SolR process. Do you know how difficult that would be to implement?
It's possible and easy to do with Solr since version 1.3 using a 'multicore' setup. On the other side, if you have a farm of Koha instances using a Solr backend you can scale up, if necessary, by deploying multiple Solr servers. So you could have for example, 1000 Koha instance running on 10 hosts and using one multicore Solr deployed on 3 servers. -- Frédéric
That is very encouraging, then. Thank you for clarifying. Clay 2010/11/11 Frédéric Demians <frederic@tamil.fr>
A possible workaround for this problem would be if multiple Koha instances' records could be stored and queried discretely from within a single SolR process. Do you know how difficult that would be to implement?
It's possible and easy to do with Solr since version 1.3 using a 'multicore' setup. On the other side, if you have a farm of Koha instances using a Solr backend you can scale up, if necessary, by deploying multiple Solr servers. So you could have for example, 1000 Koha instance running on 10 hosts and using one multicore Solr deployed on 3 servers. -- Frédéric
LAURENT Henri-Damien schreef op ma 04-10-2010 om 10:10 [+0200]:
BibLibre investigated in a catalogue based on solr.
Not sure if this is known, but I just saw it: http://www.indexdata.com/blog/2010/09/solr-support-zoom-pazpar2-and-masterke... "...we have just completed a project to add support for SOLR targets in the ZOOM API implementation in the YAZ library. So YAZ now supports Z39.50, SRU/SRW 1.x and the SOLR API." -- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5957 6D23 8B16 EFAB FEF8 7175 14D3 6485 A99C EB6D
Reply inline: On Tue, November 30, 2010 21:56, Robin Sheat wrote:
LAURENT Henri-Damien schreef op ma 04-10-2010 om 10:10 [+0200]:
BibLibre investigated in a catalogue based on solr.
Not sure if this is known, but I just saw it:
http://www.indexdata.com/blog/2010/09/solr-support-zoom-pazpar2-and-masterke...
"...we have just completed a project to add support for SOLR targets in the ZOOM API implementation in the YAZ library. So YAZ now supports Z39.50, SRU/SRW 1.x and the SOLR API."
The fact that some Index Data products recently added some Solr/Lucene support is helpful and has been cited previously but there are significant limitations. I have been investigating deeply examining source code and communicating with the developers including Sebastian Hammer at Index Data. All the options seem to fall short for providing a sufficiently comparable feature set to what we have with Zebra as a Z39.50/SRU server somewhere beyond matching query indexes with Solr/Lucene indexes. Even something as simple as distinguishing a phrase query from a word list query sent to a Z39.50/SRU server for rewriting as an appropriate Solr/Lucene query seems to require more work to develop judging by not finding appropriate source code. I am expecting explicit confirmation from a couple of people but that finding is part of my findings thus far. The source code may be sufficiently informative but I want a little more feedback from Index Data and Knowledge Integration before posting a comparison of Z39.50/SRU server options in the BibLibre Solr/Lucene RFC. When I last communicated with Ian Ibbotson at Knowledge Integration about whether he had received my last set of questions about JZKit as I had no reply, I told him that there was no hurry for an answer as long as he intended to reply when he had time. If we are having a meeting about the issue soon, I should encourage him to answer now or I will rely upon my findings in the source code. Ian Ibbotson has been at least as helpful in enabling me to think better about options from Index Data as he has been for options from Knowledge Integration. [...] Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783
participants (23)
-
Chris Cormack -
Chris Cormack -
Christopher Nighswonger -
Clay Fouts -
Colin Campbell -
Fouts, Clay
-
Frederic Demians -
Frédéric Demians
-
Galen Charlton -
glawson -
glawson＠rhcl.org
-
Ian Walls -
LAURENT Henri-Damien -
LAURENT Henri-Damien -
LAURENT Henri-Damien -
MJ Ray -
MJ Ray -
Paul Poulain -
Robin Sheat -
Salvador Zaragoza Rubio -
Thomas Dukleth -
Thomas Krichel -
Zeno Tajoli