Massime import slow (around 800 record in 1 minute).
Hi to all, I'm finding a quite slow import into a Koha DB with bulkmarcimport.pl. It imports only 800 record in 1 minute. With top: 90-95 % bulkmarcimport.pl 5 % mysql apache, zebra, indexing are down. I have 2 CPU so total CPU load is around 50% Full of RAM, 8 Gb, used 4 Gb. Option used: -b -commit 1000 -file /patth of file Do you have suggestions on how to speed the import ? Bye Zeno Tajoli -- Dr. Zeno Tajoli Servizi Innovativi -- Automazione Biblioteche z.tajoli@cineca.it fax +39 02 2135520 CINECA - Sede operativa di Segrate
Tajoli Zeno wrote:
I'm finding a quite slow import into a Koha DB with bulkmarcimport.pl. It imports only 800 record in 1 minute. With top:
90-95 % bulkmarcimport.pl 5 % mysql
apache, zebra, indexing are down.
Down the list rather than down=not-running, I hope! [...]
Do you have suggestions on how to speed the import ?
Not without stepping through the code in a debugger or using a profiler on it. Is the bulkmarcimport a one-time process (it'll be quicker to wait for it, rather than to spend lots of time speeding it up) or something you do often? Imports aren't the speediest task but I've not seen it that slow. If you could to make the import file available somehow (please not email attachment and probably not public unless you "own" all the data in it), maybe someone on IRC or otherwise can try it on a test server for you and how fast it goes, or spot any obvious problems with the data. Regards, -- MJ Ray (slef), member of www.software.coop, a for-more-than-profit co-op http://koha-community.org supporter, web and library systems developer. In My Opinion Only: see http://mjr.towers.org.uk/email.html Available for hire (including development) at http://www.software.coop/
Hi to all, Il 16/02/2015 13:57, MJ Ray ha scritto:
Tajoli Zeno wrote:
I'm finding a quite slow import into a Koha DB with bulkmarcimport.pl. It imports only 800 record in 1 minute. With top:
90-95 % bulkmarcimport.pl 5 % mysql
apache, zebra, indexing are down.
Down the list rather than down=not-running, I hope!
no MJ Ray, down=not-running. I imported 300.000 biblio records in a empty Koha. At the end the speed is 1200 record in 1 minute as average. Records without items are much more fast. I'm trying to understand better the situation. Bye Zeno Tajoli -- Dr. Zeno Tajoli Servizi Innovativi -- Automazione Biblioteche z.tajoli@cineca.it fax +39 02 2135520 CINECA - Sede operativa di Segrate
I too have noticed recent versions of koha are very slow on the bulkmarcimport. Both dev and package install. Both new installs (3.18.02.000). The larger system (16Gb) took about 10 hours bulkmarcimport for about 400,000 records being about 667 per minute. The smaller is progressing at a similar rate. I did notice that the zebraqueue table was building up as the load progressed whereas previously I was sure I had to perform a rebuild_zebra afterwards. I wonder if additional processing has been added to bulkmarcimport which might benefit smaller loads by flagging for zebra updates, but has a major impact on larger imports. If so, would it be better to have a command-line switch to avoid the extra overheads? I hope this helps the discussion. All the best. Ian On 16/02/2015 13:32, Tajoli Zeno wrote:
Hi to all, Il 16/02/2015 13:57, MJ Ray ha scritto:
Tajoli Zeno wrote:
I'm finding a quite slow import into a Koha DB with bulkmarcimport.pl. It imports only 800 record in 1 minute. With top:
90-95 % bulkmarcimport.pl 5 % mysql
apache, zebra, indexing are down.
Down the list rather than down=not-running, I hope!
no MJ Ray, down=not-running. I imported 300.000 biblio records in a empty Koha. At the end the speed is 1200 record in 1 minute as average. Records without items are much more fast.
I'm trying to understand better the situation.
Bye Zeno Tajoli
-- Ian Bays Director of Projects, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7774 995297 (mobile) +44 (0) 800 756 6384 (fax) skype: ian.bays email: ian.bays@ptfs-europe.com
commit daf2ebc4f5aeaf3a73e11915a17a2a81eafc5cb9 Date: Mon Oct 21 16:17:57 2013 -0700 Bug 11096: support the retrieval of large MARCXML records Could impact the speed if you use the --match parameter. 2015-02-16 16:32 GMT+01:00 Ian Bays <ian.bays@ptfs-europe.com>:
I too have noticed recent versions of koha are very slow on the bulkmarcimport. Both dev and package install. Both new installs (3.18.02.000).
The larger system (16Gb) took about 10 hours bulkmarcimport for about 400,000 records being about 667 per minute. The smaller is progressing at a similar rate.
I did notice that the zebraqueue table was building up as the load progressed whereas previously I was sure I had to perform a rebuild_zebra afterwards.
I wonder if additional processing has been added to bulkmarcimport which might benefit smaller loads by flagging for zebra updates, but has a major impact on larger imports.
If so, would it be better to have a command-line switch to avoid the extra overheads?
I hope this helps the discussion.
All the best. Ian
On 16/02/2015 13:32, Tajoli Zeno wrote:
Hi to all, Il 16/02/2015 13:57, MJ Ray ha scritto:
Tajoli Zeno wrote:
I'm finding a quite slow import into a Koha DB with bulkmarcimport.pl. It imports only 800 record in 1 minute. With top:
90-95 % bulkmarcimport.pl 5 % mysql
apache, zebra, indexing are down.
Down the list rather than down=not-running, I hope!
no MJ Ray, down=not-running. I imported 300.000 biblio records in a empty Koha. At the end the speed is 1200 record in 1 minute as average. Records without items are much more fast.
I'm trying to understand better the situation.
Bye Zeno Tajoli
-- Ian Bays Director of Projects, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7774 995297 (mobile) +44 (0) 800 756 6384 (fax) skype: ian.bays email: ian.bays@ptfs-europe.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
Hi. Plain and simple command line unfortunately: kohaclone/misc/migration_tools/bulkmarcimport.pl -file filename.mrc -v -l Ian On 16/02/2015 15:49, Jonathan Druart wrote:
commit daf2ebc4f5aeaf3a73e11915a17a2a81eafc5cb9 Date: Mon Oct 21 16:17:57 2013 -0700 Bug 11096: support the retrieval of large MARCXML records
Could impact the speed if you use the --match parameter.
2015-02-16 16:32 GMT+01:00 Ian Bays <ian.bays@ptfs-europe.com>:
I too have noticed recent versions of koha are very slow on the bulkmarcimport. Both dev and package install. Both new installs (3.18.02.000).
The larger system (16Gb) took about 10 hours bulkmarcimport for about 400,000 records being about 667 per minute. The smaller is progressing at a similar rate.
I did notice that the zebraqueue table was building up as the load progressed whereas previously I was sure I had to perform a rebuild_zebra afterwards.
I wonder if additional processing has been added to bulkmarcimport which might benefit smaller loads by flagging for zebra updates, but has a major impact on larger imports.
If so, would it be better to have a command-line switch to avoid the extra overheads?
I hope this helps the discussion.
All the best. Ian
On 16/02/2015 13:32, Tajoli Zeno wrote:
Hi to all, Il 16/02/2015 13:57, MJ Ray ha scritto:
Tajoli Zeno wrote:
I'm finding a quite slow import into a Koha DB with bulkmarcimport.pl. It imports only 800 record in 1 minute. With top:
90-95 % bulkmarcimport.pl 5 % mysql
apache, zebra, indexing are down.
Down the list rather than down=not-running, I hope!
no MJ Ray, down=not-running. I imported 300.000 biblio records in a empty Koha. At the end the speed is 1200 record in 1 minute as average. Records without items are much more fast.
I'm trying to understand better the situation.
Bye Zeno Tajoli
-- Ian Bays Director of Projects, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7774 995297 (mobile) +44 (0) 800 756 6384 (fax) skype: ian.bays email: ian.bays@ptfs-europe.com
_______________________________________________ Koha-devel mailing list Koha-devel@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
-- Ian Bays Director of Projects, PTFS Europe Limited Content Management and Library Solutions +44 (0) 800 756 6803 (phone) +44 (0) 7774 995297 (mobile) +44 (0) 800 756 6384 (fax) skype: ian.bays email: ian.bays@ptfs-europe.com
Le 10/02/2015 10:13, Tajoli Zeno a écrit :
Hi to all,
I'm finding a quite slow import into a Koha DB with bulkmarcimport.pl. It imports only 800 record in 1 minute. Could it be an effect of the performance issue we see in 3.18 (http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13690) ? With top:
90-95 % bulkmarcimport.pl 5 % mysql
apache, zebra, indexing are down. I have 2 CPU so total CPU load is around 50% Full of RAM, 8 Gb, used 4 Gb.
Option used: -b -commit 1000 -file /patth of file
Do you have suggestions on how to speed the import ?
Bye Zeno Tajoli
-- Paul Poulain, Associé-gérant / co-owner BibLibre, expert du logiciel libre pour les bibliothèques BibLibre, Open Source software for libraries expert
participants (5)
-
Ian Bays -
Jonathan Druart -
MJ Ray -
Paul Poulain -
Tajoli Zeno