[Koha-devel] Massime import slow (around 800 record in 1 minute).

[Jonathan Druart]

commit daf2ebc4f5aeaf3a73e11915a17a2a81eafc5cb9
Date:   Mon Oct 21 16:17:57 2013 -0700
    Bug 11096: support the retrieval of large MARCXML records

Could impact the speed if you use the --match parameter.

2015-02-16 16:32 GMT+01:00 Ian Bays <ian.bays at ptfs-europe.com>:
> I too have noticed recent versions of koha are very slow on the
> bulkmarcimport.
> Both dev and package install.  Both new installs (3.18.02.000).
>
> The larger system (16Gb) took about 10 hours bulkmarcimport for about
> 400,000 records being about 667 per minute.
> The smaller is progressing at a similar rate.
>
> I did notice that the zebraqueue table was building up as the load
> progressed whereas previously I was sure I had to perform a rebuild_zebra
> afterwards.
>
> I wonder if additional processing has been added to bulkmarcimport which
> might benefit smaller loads by flagging for zebra updates, but has a major
> impact on larger imports.
>
> If so, would it be better to have a command-line switch to avoid the extra
> overheads?
>
> I hope this helps the discussion.
>
> All the best.
> Ian
>
> On 16/02/2015 13:32, Tajoli Zeno wrote:
>>
>> Hi to all,
>> Il 16/02/2015 13:57, MJ Ray ha scritto:
>>>
>>> Tajoli Zeno wrote:
>>>>
>>>> I'm finding a quite slow import into a Koha DB with bulkmarcimport.pl.
>>>> It imports only 800 record in 1 minute.
>>>> With top:
>>>>
>>>> 90-95 % bulkmarcimport.pl
>>>> 5 % mysql
>>>>
>>>> apache, zebra, indexing are down.
>>>
>>>
>>> Down the list rather than down=not-running, I hope!
>>
>>
>> no MJ Ray, down=not-running.
>> I imported 300.000 biblio records in a empty Koha.
>> At the end the speed is 1200 record in 1 minute as average.
>> Records without items are much more fast.
>>
>> I'm trying to understand better the situation.
>>
>> Bye
>> Zeno Tajoli
>>
>
>
> --
> Ian Bays
> Director of Projects, PTFS Europe Limited
> Content Management and Library Solutions
> +44 (0) 800 756 6803 (phone)
> +44 (0) 7774 995297 (mobile)
> +44 (0) 800 756 6384 (fax)
> skype: ian.bays
> email: ian.bays at ptfs-europe.com
>
> _______________________________________________
> Koha-devel mailing list
> Koha-devel at lists.koha-community.org
> http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : http://www.koha-community.org/
> git : http://git.koha-community.org/
> bugs : http://bugs.koha-community.org/