[Bug 43787] New: Adding orders from a staged file ( addorderiso2709.pl ) can use gigabytes of memory if there are many funds
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43787 Bug ID: 43787 Summary: Adding orders from a staged file ( addorderiso2709.pl ) can use gigabytes of memory if there are many funds Initiative type: --- Sponsorship --- status: Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: Acquisitions Assignee: koha-bugs@lists.koha-community.org Reporter: kyle@bywatersolutions.com QA Contact: testopia@bugs.koha-community.org Target Milestone: --- The page for adding orders to a basket from a staged file has a fund dropdown for each record, and another for each item when MarcItemFieldsToOrder is used. Each dropdown lists every fund from every budget, inactive ones included, and the template loops that build them are so deeply indented that most of the page is whitespace. With about 1,000 funds ( not difficult for a large system that rolls over funds anually ) and a staged file of 140 records and 287 items, the page is 367 MB, 88% of it whitespace, and the Plack worker that renders it peaks at 4.5 GB of memory. To recreate: 1) Create 1,000 funds in an inactive budget 2) Set MarcItemFieldsToOrder to: homebranch: 952$a holdingbranch: 952$b itype: 952$y 3) Stage a file of 140 records with items from Add to basket > From a new file 4) Click "Add staged files to basket" 5) Note the size of the page and the memory used by the Plack worker -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43787 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |kyle@bywatersolutions.com |ity.org | -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43787 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andrew@bywatersolutions.com | |, | |kyle@bywatersolutions.com, | |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43787 --- Comment #1 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 207831 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=207831&action=edit Test order record for use with koha-testing-docker -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43787 Kyle M Hall (khall) <kyle@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |Needs Signoff -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43787 --- Comment #2 from Kyle M Hall (khall) <kyle@bywatersolutions.com> --- Created attachment 207832 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=207832&action=edit Bug 43787: Reduce the size of the html rendered by addorderiso2709.pl via collapse TT filter The page for adding orders to a basket from a staged file has a fund dropdown for each record, and another for each item when MarcItemFieldsToOrder is used. Each dropdown lists every fund from every budget, including inactive ones that are only hidden by JavaScript. The template loops that build those options are deeply indented and nothing removes the whitespace, so each option carries hundreds of bytes of spaces and newlines. With about 1,000 funds and a file of 140 records and 281 items the page is 349 MB, 88% of it whitespace, and the Starman worker that renders it peaks at 4.3 GB of memory. Wrapping those blocks in a collapse filter from template toolkit cuts the page size down to 56 MB, with worker RAM usage dropping to 912 MB! Test Plan: 1) Create 1,000 funds in an inactive budget: INSERT INTO aqbudgetperiods ( budget_period_startdate, budget_period_enddate, budget_period_active, budget_period_description, budget_period_total ) VALUES ( '2020-01-01', '2020-12-31', 0, 'Old test budget', 0 ); INSERT INTO aqbudgets ( budget_period_id, budget_code, budget_name, budget_amount ) WITH RECURSIVE seq ( n ) AS ( SELECT 1 UNION ALL SELECT n + 1 FROM seq WHERE n < 1000 ) SELECT LAST_INSERT_ID(), CONCAT( 'OLD', n ), CONCAT( 'Old test fund number ', n ), 1000 FROM seq; 2) Set MarcItemFieldsToOrder to: homebranch: 952$a holdingbranch: 952$b itype: 952$y 3) Download the attached order_test.mrc ( 140 records with 281 items ) 4) Create a basket, choose Add to basket > From a new file, stage order_test.mrc 5) Click "Add staged files to basket" 6) Note the page is about 350 MB in your browser's network tab and takes a long time to load 7) Apply this patch 8) Restart all the things! 9) Reload the page, note it is now about 55 MB! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43787 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43787 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #207832|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43787 --- Comment #3 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 207896 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=207896&action=edit Bug 43787: Reduce the size of the html rendered by addorderiso2709.pl via collapse TT filter The page for adding orders to a basket from a staged file has a fund dropdown for each record, and another for each item when MarcItemFieldsToOrder is used. Each dropdown lists every fund from every budget, including inactive ones that are only hidden by JavaScript. The template loops that build those options are deeply indented and nothing removes the whitespace, so each option carries hundreds of bytes of spaces and newlines. With about 1,000 funds and a file of 140 records and 281 items the page is 349 MB, 88% of it whitespace, and the Starman worker that renders it peaks at 4.3 GB of memory. Wrapping those blocks in a collapse filter from template toolkit cuts the page size down to 56 MB, with worker RAM usage dropping to 912 MB! Test Plan: 1) Create 1,000 funds in an inactive budget: INSERT INTO aqbudgetperiods ( budget_period_startdate, budget_period_enddate, budget_period_active, budget_period_description, budget_period_total ) VALUES ( '2020-01-01', '2020-12-31', 0, 'Old test budget', 0 ); INSERT INTO aqbudgets ( budget_period_id, budget_code, budget_name, budget_amount ) WITH RECURSIVE seq ( n ) AS ( SELECT 1 UNION ALL SELECT n + 1 FROM seq WHERE n < 1000 ) SELECT LAST_INSERT_ID(), CONCAT( 'OLD', n ), CONCAT( 'Old test fund number ', n ), 1000 FROM seq; 2) Set MarcItemFieldsToOrder to: homebranch: 952$a holdingbranch: 952$b itype: 952$y 3) Download the attached order_test.mrc ( 140 records with 281 items ) 4) Create a basket, choose Add to basket > From a new file, stage order_test.mrc 5) Click "Add staged files to basket" 6) Note the page is about 350 MB in your browser's network tab and takes a long time to load 7) Apply this patch 8) Restart all the things! 9) Reload the page, note it is now about 55 MB! Signed-off-by: Barbara Johnson <Barbara.Johnson@bedfordtx.gov> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43787 Lisette Scheer <lisette@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- QA Contact|testopia@bugs.koha-communit |matt.blenkinsop@openfifth.c |y.org |o.uk -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org