Re: [Koha-devel] MARC record size limit
Reply inline: Original Subject: [Koha-devel] Search Engine Changes : let's get some solr On Mon, October 4, 2010 08:10, LAURENT Henri-Damien wrote: [...]
zebra is fast and embeds native z3950 server. But it has also some major drawbacks we have to cope with on our everyday life making it quite difficult to maintain.
[...]
I think that every one agrees that we have to refactor C4::Search. Indeed, query parser is not able to manage independantly all the configuration options. And usage of usmarc as internal for biblio comes with a serious limitation of 9999 bytes, which for big biblios with many items, is not enough.
How do MARC limitations on record size relate to Solr/Indexing or Zebra indexing which lacks Solr/Lucene support in the current version? How does BibLibre intend to fix the limitation on the size of bibliographic records as part of its work on record indexing and retrieval in Koha or in some parallel work.?
BibLibre investigated in a catalogue based on solr. A University in France contracted us for that development. This University is in relation with all the community here in France and solr will certainly be adopted by all the libraries France wide. We are planning to release the code on our git early spring next year and rebase on whatever Koha version will be released at that time 3.4 or 3.6.
Why ?
Solr indexes with data with HTTP. It can provide fuzzy search, search on synonyms, suggestions It can provide facet search, stemming. utf8 support is embedded. Community is really impressively reactive and numerous and efficient. And documentation is very good and exhaustive.
You can see the results on solr.biblibre.com and catalogue.solr.biblibre.com
http://catalogue.solr.biblibre.com/cgi-bin/koha/opac-search.pl?q=jean http://solr.biblibre.com/cgi-bin/koha/admin/admin-home.pl you can log there with demo/demo lgoin/password
http://solr.biblibre.com/cgi-bin/koha/solr/indexes.pl is the page where ppl can manage their indexes and links.
a) Librarians can define their own indexes, and there is a plugin that fetches data from rejected authorities and from authorised_values (that could/should have been achieved with zebra but only with major work on xslt).
b) C4/Search.pm count lines of code could be shrinked ten times. You can test from poc_solr branch on git://git.biblibre.com/koha_biblibre.git But you have to install solr.
Any feedback/idea welcome.
[...] Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783
Reply inline:
Original Subject: [Koha-devel] Search Engine Changes : let's get some solr
On Mon, October 4, 2010 08:10, LAURENT Henri-Damien wrote:
[...]
zebra is fast and embeds native z3950 server. But it has also some major drawbacks we have to cope with on our everyday life making it quite difficult to maintain.
[...]
I think that every one agrees that we have to refactor C4::Search. Indeed, query parser is not able to manage independantly all the configuration options. And usage of usmarc as internal for biblio comes with a serious limitation of 9999 bytes, which for big biblios with many items, is not enough.
How do MARC limitations on record size relate to Solr/Indexing or Zebra indexing which lacks Solr/Lucene support in the current version? Koha is now using iso2709 returned from zebra in order to display result
Le 12/10/2010 14:48, Thomas Dukleth a écrit : lists. Problem is that if zebra is returning only part of the biblio and/or MARC::Record is not able to parse the whole data then the biblio is not displayed. We have biblio records which contains more than 1000 items. And MARC::Record/MARC::File::XML fails to parse that. So this is a real issue.
How does BibLibre intend to fix the limitation on the size of bibliographic records as part of its work on record indexing and retrieval in Koha or in some parallel work.?
Solr/Lucene can return indexes and thoses be used in order to display desired data or we could also do the same as we do with zebra : - store the data record (Format could be iso2709 or marcxml or YAML) - use that for display. Or we could use GetBiblio in order to get the data from database. Problem now would be the fact that storing xml in database is not really optimal for process. -- Henri-Damien LAURENT
Reply inline: On Tue, October 12, 2010 16:20, LAURENT Henri-Damien wrote:
Le 12/10/2010 14:48, Thomas Dukleth a écrit :
Reply inline:
Original Subject: [Koha-devel] Search Engine Changes : let's get some solr
On Mon, October 4, 2010 08:10, LAURENT Henri-Damien wrote:
[...]
I think that every one agrees that we have to refactor C4::Search. Indeed, query parser is not able to manage independantly all the configuration options. And usage of usmarc as internal for biblio comes with a serious limitation of 9999 bytes, which for big biblios with many items, is not enough.
How do MARC limitations on record size relate to Solr/Indexing or Zebra indexing which lacks Solr/Lucene support in the current version? Koha is now using iso2709 returned from zebra in order to display result lists.
I recall that having Zebra return ISO2709, MARC communications format, records had the supposed advantage of faster response time from Zebra.
Problem is that if zebra is returning only part of the biblio and/or MARC::Record is not able to parse the whole data then the biblio is not displayed. We have biblio records which contains more than 1000 items. And MARC::Record/MARC::File::XML fails to parse that.
So this is a real issue.
Ultimately, we need a specific solution to various problems arising from storing holdings directly in the MARC bibliographic records.
How does BibLibre intend to fix the limitation on the size of bibliographic records as part of its work on record indexing and retrieval in Koha or in some parallel work.?
Solr/Lucene can return indexes and thoses be used in order to display desired data or we could also do the same as we do with zebra : - store the data record (Format could be iso2709 or marcxml or YAML) - use that for display.
If using ISO 2709, MARC communications format, how would the problem of excess record size be addressed?
Or we could use GetBiblio in order to get the data from database. Problem now would be the fact that storing xml in database is not really optimal for process.
I like the idea of using YAML for some purposes. As you state, previous testing showed that returning every record in a large result set from the SQL database was very inefficient as compared to using the records as part of the response from the index server. Is there any practical way of sufficiently improving the efficiency of accessing a large set of records from the SQL database? How much might retrieving and parsing YAML records from the database help? I can imagine using XSLT to pre-process MARCXML records into an appropriate format, such YAML with embedded HTML, pure HTML, or whatever is needed embedded for a particular purpose and storing the pre-processed records in appropriate special purpose columns. Real time parsing would be minimised. The OPAC result set display might use biblioitems.recordOPACDisplayBrief. The standard single record view might use biblioitems.recordOPACDisplayDetail. An ISBD card view might use biblioitems.recordOPACDisplayISBD. [...] Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783
I did some (very limited) testing on storing and retrieving MARC in YAML. The results were not encouraging. IIRC, I just did a direct conversion of the MARC::Record object into YAML and back. Perhaps there's a way to optimize the formatting that would improve performance, but my testing showed sometimes even worse performance than XML. MARCXML is a performance killer at this point, but there's no other apparent way to handle large bib records. The parsing is the issue, not the data transfer load. Perhaps cached BSON-formatted MARC::Record objects are a way out of this. Clay On Tue, Oct 12, 2010 at 11:45 AM, Thomas Dukleth <kohadevel@agogme.com>wrote:
Reply inline:
On Tue, October 12, 2010 16:20, LAURENT Henri-Damien wrote:
Le 12/10/2010 14:48, Thomas Dukleth a écrit :
Reply inline:
Original Subject: [Koha-devel] Search Engine Changes : let's get some solr
On Mon, October 4, 2010 08:10, LAURENT Henri-Damien wrote:
[...]
I think that every one agrees that we have to refactor C4::Search. Indeed, query parser is not able to manage independantly all the configuration options. And usage of usmarc as internal for biblio comes with a serious limitation of 9999 bytes, which for big biblios with many items, is not enough.
How do MARC limitations on record size relate to Solr/Indexing or Zebra indexing which lacks Solr/Lucene support in the current version? Koha is now using iso2709 returned from zebra in order to display result lists.
I recall that having Zebra return ISO2709, MARC communications format, records had the supposed advantage of faster response time from Zebra.
Problem is that if zebra is returning only part of the biblio and/or MARC::Record is not able to parse the whole data then the biblio is not displayed. We have biblio records which contains more than 1000 items. And MARC::Record/MARC::File::XML fails to parse that.
So this is a real issue.
Ultimately, we need a specific solution to various problems arising from storing holdings directly in the MARC bibliographic records.
How does BibLibre intend to fix the limitation on the size of bibliographic records as part of its work on record indexing and retrieval in Koha or in some parallel work.?
Solr/Lucene can return indexes and thoses be used in order to display desired data or we could also do the same as we do with zebra : - store the data record (Format could be iso2709 or marcxml or
YAML)
- use that for display.
If using ISO 2709, MARC communications format, how would the problem of excess record size be addressed?
Or we could use GetBiblio in order to get the data from database. Problem now would be the fact that storing xml in database is not really optimal for process.
I like the idea of using YAML for some purposes.
As you state, previous testing showed that returning every record in a large result set from the SQL database was very inefficient as compared to using the records as part of the response from the index server.
Is there any practical way of sufficiently improving the efficiency of accessing a large set of records from the SQL database? How much might retrieving and parsing YAML records from the database help?
I can imagine using XSLT to pre-process MARCXML records into an appropriate format, such YAML with embedded HTML, pure HTML, or whatever is needed embedded for a particular purpose and storing the pre-processed records in appropriate special purpose columns. Real time parsing would be minimised. The OPAC result set display might use biblioitems.recordOPACDisplayBrief. The standard single record view might use biblioitems.recordOPACDisplayDetail. An ISBD card view might use biblioitems.recordOPACDisplayISBD.
[...]
Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
I did some (very limited) testing on storing and retrieving MARC in YAML. The results were not encouraging. IIRC, I just did a direct conversion of the MARC::Record object into YAML and back. Perhaps there's a way to optimize the formatting that would improve performance, but my testing showed sometimes even worse performance than XML.
Did you use YAML or YAML::XS? My tests with YAML::XS shows a very significant improvement with YAML: see attached file. Of course, we should define an serialization format independent from MARC::Record object if we don't want to break the process when MARC::Record internal data structure ever change.
MARCXML is a performance killer at this point, but there's no other apparent way to handle large bib records. The parsing is the issue, not the data transfer load. Perhaps cached BSON-formatted MARC::Record objects are a way out of this.
Benchmark should be done with all available serialization formats. We also could implement serialization/deserialization logic directly into MARC::Record library, as ISO2709 and XML format, in order gain control. -- Frédéric
That is substantially faster, and more in line with my expectation of what the performance difference would be. I had thought I was using the XS module, but clearly something was amiss! This is good news. I would think implementing within MARC::Record would be the winner between those two approaches, rather than coming up with an intermediary format. However, I had been imagining utilizing the YAML for caching rather than as persistent data, making the accommodation of internal changes to MARC::Record a matter of flushing the cache rather than regenerating the serialized records. Cheers, Clay 2010/10/26 Frederic Demians <frederic@tamil.fr>
I did some (very limited) testing on storing and retrieving MARC in YAML.
The results were not encouraging. IIRC, I just did a direct conversion of the MARC::Record object into YAML and back. Perhaps there's a way to optimize the formatting that would improve performance, but my testing showed sometimes even worse performance than XML.
Did you use YAML or YAML::XS? My tests with YAML::XS shows a very significant improvement with YAML: see attached file. Of course, we should define an serialization format independent from MARC::Record object if we don't want to break the process when MARC::Record internal data structure ever change.
MARCXML is a performance killer at this point, but there's no other
apparent way to handle large bib records. The parsing is the issue, not the data transfer load. Perhaps cached BSON-formatted MARC::Record objects are a way out of this.
Benchmark should be done with all available serialization formats.
We also could implement serialization/deserialization logic directly into MARC::Record library, as ISO2709 and XML format, in order gain control. -- Frédéric
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
Reply inline: On Tue, October 26, 2010 22:01, Fouts, Clay wrote:
I did some (very limited) testing on storing and retrieving MARC in YAML. The results were not encouraging. IIRC, I just did a direct conversion of the MARC::Record object into YAML and back. Perhaps there's a way to optimize the formatting that would improve performance, but my testing showed sometimes even worse performance than XML.
I had not suggested YAML as a prospective data format for simple real time conversion to and from MARC or MARCXML. The potential value which I see in YAML is for storing data types appropriately for special purposes where a record would exist in whatever primary form in Koha and also exist in a completely transformed forms for indexing, display, record exchange, etc. Storing record data normalised for a particular purpose as strings, numeric values, ordered lists, arrays, etc. where appropriate after parsing it from the original MARC record strings is very different from merely storing MARC in a different record syntax. YAML provides data typing which neither MARC nor XML do. MARC whether in MARC communications format (ISO 2709) or MARCXML does not provide sufficient normalisation for many purposes. In considering YAML, I do not exclude the possibility that creating a special normalised XML record format which encodes data type in attributes etc. may be more easily supported than YAML or better in some other way However, XML can be easily embedded in YAML.
MARCXML is a performance killer at this point, but there's no other apparent way to handle large bib records. The parsing is the issue, not the data transfer load. Perhaps cached BSON-formatted MARC::Record objects are a way out of this.
[...] Thomas Dukleth Agogme 109 E 9th Street, 3D New York, NY 10003 USA http://www.agogme.com +1 212-674-3783
Le 13/10/2010 07:45, Thomas Dukleth a écrit :
XSLT to pre-process MARCXML records into an appropriate format, such YAML with embedded HTML, pure HTML, or whatever is needed embedded for a particular purpose and storing the pre-processed records in appropriate special purpose columns. Real time parsing would be minimised. The OPAC result set display might use biblioitems.recordOPACDisplayBrief. The standard single record view might use biblioitems.recordOPACDisplayDetail. An ISBD card view might use biblioitems.recordOPACDisplayISBD.
Adding more fields in the database is a noway I think. Since it would make DB grow and would just be kind of caching results. Maybe we could just use function and caching results. Maybe we could use disk space for that. Synching and update process is ressource demanding enough. So I definitely think that we should use Memcached for that, provided the mighty cautions Clay did in the Thread on DBIx::Class. -- Henri-Damien LAURENT
participants (4)
-
Fouts, Clay -
Frederic Demians -
LAURENT Henri-Damien -
Thomas Dukleth