[Bug 16855] New: Poor performance due to high overhead of SQL call in export.pl
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16855 Bug ID: 16855 Summary: Poor performance due to high overhead of SQL call in export.pl Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Tools Assignee: gmcharlt@gmail.com Reporter: nick@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org Export.pl uses Koha::Biblioitems->search() to retrieve biblionumbers for the export process. The SQL code generated by this does a full dump of the the biblioitems table including the marcxml for each record. This table can be quite large and on smaller servers can actually cause debian to fire oom-killer on the process. As the results of the search are only used to push biblionumbers into an array it might make better sense to do a direct SQL query and not retrieve the marcxml column -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16855 Nick Clemens <nick@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Patch complexity|--- |Trivial patch Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16855 --- Comment #1 from Nick Clemens <nick@bywatersolutions.com> --- Created attachment 53110 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=53110&action=edit Bug 16855 - Poor performance due to high overhead of SQL call in export.pl This patch eliminates all columns retrieved in the biblioitems query except for biblionumber. To test: 1 - Go to tools->Export data 2 - Export using various filters and note you get expected records 3 - Apply patch 4 - Ensure exported results match results before patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16855 Srdjan Jankovic <srdjan@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16855 Srdjan Jankovic <srdjan@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #53110|0 |1 is obsolete| | --- Comment #2 from Srdjan Jankovic <srdjan@catalyst.net.nz> --- Created attachment 53122 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=53122&action=edit [SIGNED-OFF] Bug 16855 - Poor performance due to high overhead of SQL call in export.pl This patch eliminates all columns retrieved in the biblioitems query except for biblionumber. To test: 1 - Go to tools->Export data 2 - Export using various filters and note you get expected records 3 - Apply patch 4 - Ensure exported results match results before patch Signed-off-by: Srdjan <srdjan@catalyst.net.nz> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16855 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16855 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #53122|0 |1 is obsolete| | --- Comment #3 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Created attachment 53138 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=53138&action=edit Bug 16855 - Poor performance due to high overhead of SQL call in export.pl This patch eliminates all columns retrieved in the biblioitems query except for biblionumber. To test: 1 - Go to tools->Export data 2 - Export using various filters and note you get expected records 3 - Apply patch 4 - Ensure exported results match results before patch Signed-off-by: Srdjan <srdjan@catalyst.net.nz> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16855 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@bugs.koha-c | |ommunity.org --- Comment #4 from Jonathan Druart <jonathan.druart@bugs.koha-community.org> --- Quick benchmark using: use Koha::Biblioitems; my @record_ids; my $biblioitems = Koha::Biblioitems->search( {}, { join => 'items', columns => 'biblionumber' } ); while ( my $biblioitem = $biblioitems->next ) { push @record_ids, $biblioitem->biblionumber; } and use Koha::Biblioitems; my @record_ids; my $biblioitems = Koha::Biblioitems->search( {}, { join => 'items' } ); while ( my $biblioitem = $biblioitems->next ) { push @record_ids, $biblioitem->biblionumber; } gives 1.4s vs 1.7s with 8600 items. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16855 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|gmcharlt@gmail.com |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16855 Jonathan Druart <jonathan.druart@bugs.koha-community.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Depends on| |14722 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14722 [Bug 14722] Refactor the catalogue export tool (command-line tools/export.pl does not work anymore. Use misc/export_records.pl instead) -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16855 Kyle M Hall <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to Master CC| |kyle@bywatersolutions.com --- Comment #5 from Kyle M Hall <kyle@bywatersolutions.com> --- Pushed to master for Koha 16.11,t thanks Nick! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16855 Frédéric Demians <frederic@tamil.fr> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |frederic@tamil.fr Status|Pushed to Master |Pushed to Stable --- Comment #6 from Frédéric Demians <frederic@tamil.fr> --- Pushed in 16.05. Will be in 16.05.02. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16855 Julian Maurice <julian.maurice@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |julian.maurice@biblibre.com --- Comment #7 from Julian Maurice <julian.maurice@biblibre.com> --- Patch pushed to 3.22.x, will be in 3.22.10 -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org