[Koha-zebra] Re: Import Speed

[Mike Taylor]

> Date: Thu, 02 Mar 2006 14:07:04 -0500
> From: Sebastian Hammer <quinn at indexdata.com>
>
>> But -- ??  What magic does the command-line import have access to
>> that ZOOM update doesn't?  Clearly it's using some kind of
>> in-memory caching to hugely reduce the frequency of disk-writes,
>> but why shouldn't that also be used the doing a ZOOM update?  Isn't
>> that (part of) the purpose of delaying the "commit" call?  If not,
>> then we need to add $conn->option("updateCacheSize" =>
>> 100*1024*1024)
>
> 'commit' has nearly nothing to do with it.

Right.  I now see that I was envisaging commit as being (in part)
analogous to the "index" operation that you posit below.

> It would be much better if we had a new stage between the updating
> of records and the commit... something to allow us to transfer a
> large number of records (preferably more than one per operation to
> cut down on the round-trip traffic), THEN index them, THEN commit
> the changes.

Precisely.

> Then we'd be able to do remote updates as efficiently as we can do
> them locally..
> 
> Something like
> 
> 1. Update, update, update, update.....
> 2. Index
> 3. Commit

Or, more often:

	update, update, update ...
	index
	update, update, update ...
	index
	update, update, update ...
	index
	...
	commit

> Mike, you can speak to Adam about this over lunch if you get a
> chance..  it is possible that I misrepresent what happens -- but
> this reflects my understanding.

I'll see what I can do.

 _/|_	 ___________________________________________________________________
/o ) \/  Mike Taylor  <mike at miketaylor.org.uk>  http://www.miketaylor.org.uk
)_v__/\  "[The cheese factory] beneath Covenant hung insubstantial,
	 lambent nacreous sepulchral vitriol ..." -- Mike Lessacher.