Date: Thu, 09 Feb 2006 13:25:50 -0500 From: Sebastian Hammer <quinn@indexdata.com>
I flatly refuse to believe that the communication layer is responsible for a slow-down by a factor of 16, so something else is going on here. My best guess is that "zebraidx update" is making use of caching mechanisms that ZOOM's update requests are not benefiting from. There may be a way to have ZOOM request that caching: Adam will be able to tell us.
I don't particularly refuse to believe the factor 16. Network overheads are very expensive.
C'mon, dude. We're adding and indexing records here. A typical MARC record might contain say 30 indexable words, so naively you're doing 30 seeks and writes. Suppose caching reduces that by a factor of ten. Still -- you're seeking and writing multiple times. No in-memory copy (which is all the TCP/IP socket write is) is going to come close to using that much time. As always we should benchmark this rather than just spouting opinions, but I bet you 422 trillion Canadian dollars that switching to a Unix-domain socket makes very little difference indeed. (If the client and server were on different machines, I would be less disinclined to swallow your hypothesis.)
I think you're only transfering one record at a time, right? That's bad for network latency in itself, but I dunno if either the client API or the server will let us do more at a time.. it might be worth considering, because adding more records at a time is generally a lot faster (per record) than adding single records.
Yes, this would be a reasonable enhancement to make to our update API.
One thing to try, if you don't already do it, is to use a UNIX domain socket if your server and client run on the same site..
zebrasrv unix:s
Will listen on a socket called 's'... since Unix Domain sockets circumvent the network stuff and work just like pipes, they can be tons faster... I usually use this when using Zebra as an embedded engine. I'm hoping Mike's ZOOM Perl wrapper won't prevent you from using this option of the lower-layer tool.
It won't! Go right ahead, Unix-domain sockets will work fine. _/|_ ___________________________________________________________________ /o ) \/ Mike Taylor <mike@miketaylor.org.uk> http://www.miketaylor.org.uk )_v__/\ "I never make predictions and I never will" -- Paul Gascoigne.