[Koha-devel] Massime import slow (around 800 record in 1 minute).

[Ian Bays]

Hi. Plain and simple command line unfortunately:

kohaclone/misc/migration_tools/bulkmarcimport.pl -file filename.mrc -v -l

Ian
On 16/02/2015 15:49, Jonathan Druart wrote:
> commit daf2ebc4f5aeaf3a73e11915a17a2a81eafc5cb9
> Date:   Mon Oct 21 16:17:57 2013 -0700
>      Bug 11096: support the retrieval of large MARCXML records
>
> Could impact the speed if you use the --match parameter.
>
> 2015-02-16 16:32 GMT+01:00 Ian Bays <ian.bays at ptfs-europe.com>:
>> I too have noticed recent versions of koha are very slow on the
>> bulkmarcimport.
>> Both dev and package install.  Both new installs (3.18.02.000).
>>
>> The larger system (16Gb) took about 10 hours bulkmarcimport for about
>> 400,000 records being about 667 per minute.
>> The smaller is progressing at a similar rate.
>>
>> I did notice that the zebraqueue table was building up as the load
>> progressed whereas previously I was sure I had to perform a rebuild_zebra
>> afterwards.
>>
>> I wonder if additional processing has been added to bulkmarcimport which
>> might benefit smaller loads by flagging for zebra updates, but has a major
>> impact on larger imports.
>>
>> If so, would it be better to have a command-line switch to avoid the extra
>> overheads?
>>
>> I hope this helps the discussion.
>>
>> All the best.
>> Ian
>>
>> On 16/02/2015 13:32, Tajoli Zeno wrote:
>>> Hi to all,
>>> Il 16/02/2015 13:57, MJ Ray ha scritto:
>>>> Tajoli Zeno wrote:
>>>>> I'm finding a quite slow import into a Koha DB with bulkmarcimport.pl.
>>>>> It imports only 800 record in 1 minute.
>>>>> With top:
>>>>>
>>>>> 90-95 % bulkmarcimport.pl
>>>>> 5 % mysql
>>>>>
>>>>> apache, zebra, indexing are down.
>>>>
>>>> Down the list rather than down=not-running, I hope!
>>>
>>> no MJ Ray, down=not-running.
>>> I imported 300.000 biblio records in a empty Koha.
>>> At the end the speed is 1200 record in 1 minute as average.
>>> Records without items are much more fast.
>>>
>>> I'm trying to understand better the situation.
>>>
>>> Bye
>>> Zeno Tajoli
>>>
>>
>> --
>> Ian Bays
>> Director of Projects, PTFS Europe Limited
>> Content Management and Library Solutions
>> +44 (0) 800 756 6803 (phone)
>> +44 (0) 7774 995297 (mobile)
>> +44 (0) 800 756 6384 (fax)
>> skype: ian.bays
>> email: ian.bays at ptfs-europe.com
>>
>> _______________________________________________
>> Koha-devel mailing list
>> Koha-devel at lists.koha-community.org
>> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
>> website : http://www.koha-community.org/
>> git : http://git.koha-community.org/
>> bugs : http://bugs.koha-community.org/


-- 
Ian Bays
Director of Projects, PTFS Europe Limited
Content Management and Library Solutions
+44 (0) 800 756 6803 (phone)
+44 (0) 7774 995297 (mobile)
+44 (0) 800 756 6384 (fax)
skype: ian.bays
email: ian.bays at ptfs-europe.com