[Koha-devel] MARC record size limit

[Thomas Dukleth]

Reply inline:


On Tue, October 26, 2010 22:01, Fouts, Clay wrote:
> I did some (very limited) testing on storing and retrieving MARC in YAML.
> The results were not encouraging. IIRC, I just did a direct conversion of
> the MARC::Record object into YAML and back. Perhaps there's a way to
> optimize the formatting that would improve performance, but my testing
> showed sometimes even worse performance than XML.

I had not suggested YAML as a prospective data format for simple real time
conversion to and from MARC or MARCXML.

The potential value which I see in YAML is for storing data types
appropriately for special purposes where a record would exist in whatever
primary form in Koha and also exist in a completely transformed forms for
indexing, display, record exchange, etc.  Storing record data normalised
for a particular purpose as strings, numeric values, ordered lists,
arrays, etc. where appropriate after parsing it from the original MARC
record strings is very different from merely storing MARC in a different
record syntax.

YAML provides data typing which neither MARC nor XML do.  MARC whether in
MARC communications format (ISO 2709) or MARCXML does not provide
sufficient normalisation for many purposes.

In considering YAML, I do not exclude the possibility that creating a
special normalised XML record format which encodes data type in attributes
etc. may be more easily supported than YAML or better in some other way 
However, XML can be easily embedded in YAML.

>
> MARCXML is a performance killer at this point, but there's no other
> apparent
> way to handle large bib records. The parsing is the issue, not the data
> transfer load. Perhaps cached BSON-formatted MARC::Record objects are a
> way
> out of this.

[...]

Thomas Dukleth
Agogme
109 E 9th Street, 3D
New York, NY  10003
USA
http://www.agogme.com
+1 212-674-3783