That is substantially faster, and more in line with my expectation of what the performance difference would be. I had thought I was using the XS module, but clearly something was amiss! This is good news.
I would think implementing within MARC::Record would be the winner between those two approaches, rather than coming up with an intermediary format. However, I had been imagining utilizing the YAML for caching rather than as persistent data, making the accommodation of internal changes to MARC::Record a matter of flushing the cache rather than regenerating the serialized records.
Cheers,
Clay
2010/10/26 Frederic Demians
<frederic@tamil.fr>
I did some (very limited) testing on storing and retrieving MARC in YAML. The results were not encouraging. IIRC, I just did a direct conversion of the MARC::Record object into YAML and back. Perhaps there's a way to optimize the formatting that would improve performance, but my testing showed sometimes even worse performance than XML.
Did you use YAML or YAML::XS? My tests with YAML::XS shows a very significant improvement with YAML: see attached file. Of course, we should define an serialization format independent from MARC::Record object if we don't want to break the process when MARC::Record internal data structure ever change.
MARCXML is a performance killer at this point, but there's no other apparent way to handle large bib records. The parsing is the issue, not the data transfer load. Perhaps cached BSON-formatted MARC::Record objects are a way out of this.
Benchmark should be done with all available serialization formats.
We also could implement serialization/deserialization logic directly into MARC::Record library, as ISO2709 and XML format, in order gain control.
--
Frédéric
_______________________________________________
Koha-devel mailing list
Koha-devel@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel