Re: [Koha-devel] Koha performance enhancement overview
[Adding koha-devel back in] On Mon, Apr 9, 2012 at 9:37 PM, Ian Walls <koha.sekjal@gmail.com> wrote:
I'd be curious if TurboMARC might be a worthwhile compromise: the extensibility of XML, but easier to process than MARCXML.
It might be worth looking into, but it seems like the amount of work necessary just to test it in a small way would be much to high to justify for idle curiosity (though if anyone feels like doing it, I wouldn't want to discourage them!). Regards, Jared -- Jared Camins-Esakov Bibliographer, C & P Bibliography Services, LLC (phone) +1 (917) 727-3445 (e-mail) jcamins@cpbibliography.com (web) http://www.cpbibliography.com/
On 2012-04-10, at 1:57 PM, Jared Camins-Esakov wrote:
[Adding koha-devel back in]
On Mon, Apr 9, 2012 at 9:37 PM, Ian Walls <koha.sekjal@gmail.com> wrote: I'd be curious if TurboMARC might be a worthwhile compromise: the extensibility of XML, but easier to process than MARCXML.
woooh, turbomarc looks pretty cool! but, it looks like zebra does not support turbomarc, (just yet) i think any turbomarc benchmarking for Koha is moot, until zebra supports turbomarc
It might be worth looking into, but it seems like the amount of work necessary just to test it in a small way would be much to high to justify for idle curiosity (though if anyone feels like doing it, I wouldn't want to discourage them!).
Regards, Jared
Mason James schreef op di 10-04-2012 om 17:09 [+1200]:
woooh, turbomarc looks pretty cool! but, it looks like zebra does not support turbomarc, (just yet)
i think any turbomarc benchmarking for Koha is moot, until zebra supports turbomarc
Well, the rebuild_zebra script could convert it on the fly if it needed to. -- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5957 6D23 8B16 EFAB FEF8 7175 14D3 6485 A99C EB6D
On 2012-04-10, at 5:11 PM, Robin Sheat wrote:
Mason James schreef op di 10-04-2012 om 17:09 [+1200]:
woooh, turbomarc looks pretty cool! but, it looks like zebra does not support turbomarc, (just yet)
i think any turbomarc benchmarking for Koha is moot, until zebra supports turbomarc
Well, the rebuild_zebra script could convert it on the fly if it needed to.
ah, ok... so we could possibly use the turbomarc format internally within koha, and just transform it to marcxml for writing out to zebra, when needed sounds do-able :) i guess we would need to patch MARC::File::XML, to correctly parse a turbomarc record? fyi: according to this url, turbomarc parsing is 4-5 times faster that marcxml. so its definitely something to have a look at :) http://www.indexdata.com/blog/2010/05/turbomarc-faster-xml-marc-records just curious, is it worth switching back to the plain old MARC format internally for Koha, for an extra speed increase? i cant remember what the advantage was in moving to marcxml? (other than marcxml being easier to read, for a human)
2012/4/10 Mason James <mtj@kohaaloha.com>:
On 2012-04-10, at 5:11 PM, Robin Sheat wrote:
Mason James schreef op di 10-04-2012 om 17:09 [+1200]:
woooh, turbomarc looks pretty cool! but, it looks like zebra does not support turbomarc, (just yet)
i think any turbomarc benchmarking for Koha is moot, until zebra supports turbomarc
Well, the rebuild_zebra script could convert it on the fly if it needed to.
ah, ok... so we could possibly use the turbomarc format internally within koha, and just transform it to marcxml for writing out to zebra, when needed sounds do-able :)
i guess we would need to patch MARC::File::XML, to correctly parse a turbomarc record?
fyi: according to this url, turbomarc parsing is 4-5 times faster that marcxml. so its definitely something to have a look at :) http://www.indexdata.com/blog/2010/05/turbomarc-faster-xml-marc-records
just curious, is it worth switching back to the plain old MARC format internally for Koha, for an extra speed increase?
Heh, thats what this whole thread started with :) You should read back
i cant remember what the advantage was in moving to marcxml? (other than marcxml being easier to read, for a human)
When we had items in the marc, it got too big and was truncated, a limitation of the iso2709 standard. Being truncated rendered it unusable because it missed the 999 (at the end). MARCXML doesn't suffer the size limit. However, now we don't have items stored in the MARC, we could switch back, if it is indeed faster, we need to test with the missing embed items function call added back in. Chris
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
On 2012-04-10, at 8:37 PM, Chris Cormack wrote:
2012/4/10 Mason James <mtj@kohaaloha.com>:
just curious, is it worth switching back to the plain old MARC format internally for Koha, for an extra speed increase?
Heh, thats what this whole thread started with :) You should read back
i cant remember what the advantage was in moving to marcxml? (other than marcxml being easier to read, for a human)
When we had items in the marc, it got too big and was truncated, a limitation of the iso2709 standard. Being truncated rendered it unusable because it missed the 999 (at the end). MARCXML doesn't suffer the size limit.
However, now we don't have items stored in the MARC, we could switch back, if it is indeed faster, we need to test with the missing embed items function call added back in.
Chris
aaah yep, the 99999 size-limit was the reason for the switch.... thanks for the reminder Chris
Hie, I've been thinking about Koha's performance. You know that, when XSLT is used on OPAC, earch search and detail page uses an XSLT engine to transform MARCXML (and a bit more) into HTML. Could it be possible to save, for each biblio, this generated HTML and cache it on web server as static content ? This will improve the web server performance when a lot of clients are connected on OPAC. A script could be planned to update cache at some interval. It may also be plugged with zebra_queue. Regards, -- Fridolyn SOMERS fridolyn.somers@gmail.com Marsillargues - France
Le 16/04/2012 14:48, Fridolyn SOMERS a écrit :
Hie,
I've been thinking about Koha's performance.
You know that, when XSLT is used on OPAC, earch search and detail page uses an XSLT engine to transform MARCXML (and a bit more) into HTML. I think a memcached XSLTed would be easier, and as efficient. Am I right ?
-- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
Good idea. But it would need a huge quantity of RAM no ? On Mon, Apr 16, 2012 at 3:33 PM, Paul Poulain <paul.poulain@biblibre.com>wrote:
Le 16/04/2012 14:48, Fridolyn SOMERS a écrit :
Hie,
I've been thinking about Koha's performance.
You know that, when XSLT is used on OPAC, earch search and detail page uses an XSLT engine to transform MARCXML (and a bit more) into HTML. I think a memcached XSLTed would be easier, and as efficient. Am I right ?
-- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08 _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolyn SOMERS fridolyn.somers@gmail.com Marsillargues - France <fridolyn.somers@gmail.com>
Good idea. But it would need a huge quantity of RAM no ? well, it does not mean *all* records would be in the cache, but only "recently" retrieved/calculated records. In memcache, you can define how much memory you have available (64M by default, which is low for xlst-ed records), and you can also compress on
Le 16/04/2012 16:13, Fridolyn SOMERS a écrit : the fly (which require some CPU, so you must choose your poison ;-) ) -- Paul POULAIN http://www.biblibre.com Expert en Logiciels Libres pour l'info-doc Tel : (33) 4 91 81 35 08
Fridolyn SOMERS schreef op ma 16-04-2012 om 16:13 [+0200]:
But it would need a huge quantity of RAM no ?
Not nearly as much as you would think, really. And it'll let the older ones expire when things get full. Also, if you don't mind gzipping it, it'll get quite tiny. -- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5957 6D23 8B16 EFAB FEF8 7175 14D3 6485 A99C EB6D
Ok, thanks, I have to learn more about memcached. 2012/4/17 Robin Sheat <robin@catalyst.net.nz>
Fridolyn SOMERS schreef op ma 16-04-2012 om 16:13 [+0200]:
But it would need a huge quantity of RAM no ?
Not nearly as much as you would think, really. And it'll let the older ones expire when things get full.
Also, if you don't mind gzipping it, it'll get quite tiny.
-- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5957 6D23 8B16 EFAB FEF8 7175 14D3 6485 A99C EB6D
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolyn SOMERS fridolyn.somers@gmail.com Marsillargues - France <fridolyn.somers@gmail.com>
Check out redis too, very cool stuff. Chris Fridolyn SOMERS <fridolyn.somers@gmail.com> wrote: Ok, thanks, I have to learn more about memcached. 2012/4/17 Robin Sheat <robin@catalyst.net.nz> Fridolyn SOMERS schreef op ma 16-04-2012 om 16:13 [+0200]:
But it would need a huge quantity of RAM no ?
Not nearly as much as you would think, really. And it'll let the older ones expire when things get full. Also, if you don't mind gzipping it, it'll get quite tiny. -- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5957 6D23 8B16 EFAB FEF8 7175 14D3 6485 A99C EB6D _______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/ -- Fridolyn SOMERS fridolyn.somers@gmail.com Marsillargues - France
Unless we're working with a small catalogue, or one with only a few popular items, I'm not sure we'd get much benefit from caching the XSLT-processed HTML for individual biblios. Outside of those cases, I don't think a lot of people will be looking at the same records, so we're just storing info until it expires, rather than reusing it. Analysis of web traffic logs would indicate whether there are enough popular records in the OPAC for it to be worth caching. Perhaps we could do like the SilverStripe CMS does, and collect hit data, and build caches of anything that has a certain threshold... -Ian 2012/4/18 Chris Cormack <chrisc@catalyst.net.nz>
** Check out redis too, very cool stuff.
Chris
Fridolyn SOMERS <fridolyn.somers@gmail.com> wrote:
Ok, thanks, I have to learn more about memcached.
2012/4/17 Robin Sheat <robin@catalyst.net.nz>
Fridolyn SOMERS schreef op ma 16-04-2012 om 16:13 [+0200]:
But it would need a huge quantity of RAM no ?
Not nearly as much as you would think, really. And it'll let the older ones expire when things get full.
Also, if you don't mind gzipping it, it'll get quite tiny.
-- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5957 6D23 8B16 EFAB FEF8 7175 14D3 6485 A99C EB6D
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Fridolyn SOMERS fridolyn.somers@gmail.com Marsillargues - France <fridolyn.somers@gmail.com>
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Unless we're working with a small catalogue, or one with only a few popular items, I'm not sure we'd get much benefit from caching the XSLT-processed HTML for individual biblios. Outside of those cases, I don't think a lot of people will be looking at the same records, so we're just storing info until it expires, rather than reusing it.
+1 And I remind you that in C4::XSLT, items information are injected into MARCXML prior to send it to XSLT for transformation into HTML. It means that if we cache HTML representation of biblio records, we loose real-time items availability in OPAC result page. Concerning biblio records, what kill performances? [1] GetMarcBiblio multiple calls GetMarcBiblio function is used, and reused, and reused again, for the same biblionumber. The CoinS tag generation for example do that: calling GetMarcBiblio for a biblio record which has already be instantiated by another function. Code could be improve. Or very simple to do, GetMarcBiblio can be memoized (at script level without persistence). [2] MARC::Record deserialization In GetMarcBiblio (and other functions?), a MARC::Record is build based on biblioitems.marcxml field. This is very slow as already discussed. Since items are not anymore in biblio records, we don't need anymore to use the biblio recorcd XML representation, and should use biblioitems.marc field. [3] XSLT transformation Koha XSL stylesheets have to be parsed one time, in each script using them. In persistent environment, this could be cached. The biblio record is retrieved from biblioitems.marcxml, then a MARC::Record object is build (it costs), then the MARC::Record is completed with various info (authorized values, etc.), then the object is serialized in XML, then the XML document is completed with items real-time info, then the XML is sent to XSLT. We would gain a lot by caching an XML version ready to be completed with items info. This way, we wouldn't have to deserialize/serialize MARC::Record objects. Kind regards, -- Frédéric DEMIANS http://www.tamil.fr/u/fdemians.html
Hie, Frédéric, you are on my side.
It means that if we cache HTML representation of biblio records, we loose real-time items availability in OPAC result page You can't have a real-time AND performance. Can't a user have an item status old from few minutes ? It is already this with search engine, it is not real-time (=> my idea to udpate cache with Zebraqueue).
Second option is to move items infos from XSLT to template. Is it really customized by libraries ? Regards, -- Fridolyn SOMERS fridolyn.somers@gmail.com Marsillargues - France <fridolyn.somers@gmail.com> <fridolyn.somers@gmail.com>
Mason, woooh, turbomarc looks pretty cool!
but, it looks like zebra does not support turbomarc, (just yet)
Sure it does. The DOM indexing filter uses XSLT to transform arbitrary XML into Zebra index definitions. The only Zebra-related change needed is the elimination of the iso2709 option for indexing. And, of course, eliminating GRS-1 indexing, but that's a much-needed change anyway. Regards, Jared -- Jared Camins-Esakov Bibliographer, C & P Bibliography Services, LLC (phone) +1 (917) 727-3445 (e-mail) jcamins@cpbibliography.com (web) http://www.cpbibliography.com/
On 2012-04-11, at 1:14 AM, Jared Camins-Esakov wrote:
Mason,
woooh, turbomarc looks pretty cool! but, it looks like zebra does not support turbomarc, (just yet)
Sure it does. The DOM indexing filter uses XSLT to transform arbitrary XML into Zebra index definitions. The only Zebra-related change needed is the elimination of the iso2709 option for indexing. And, of course, eliminating GRS-1 indexing, but that's a much-needed change anyway.
bah, i'm always the slow one :p but i am very happy to be wrong here. turbomarc could be the perfect compromise between MARC and MARCXML for Koha. 4-5 times faster than MARCXML, but not 99999-octet size-limited like MARC
participants (9)
-
Chris Cormack -
Chris Cormack -
Fridolyn SOMERS -
Frédéric Demians -
Ian Walls -
Jared Camins-Esakov -
Mason James -
Paul Poulain -
Robin Sheat