https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23685 Bug ID: 23685 Summary: Exporting report may consume unlimited memory Change sponsored?: --- Product: Koha Version: master Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Reports Assignee: koha-bugs@lists.koha-community.org Reporter: paul@flo.org QA Contact: testopia@bugs.koha-community.org In guided_reports.pl, when exporting report results ($phase eq 'Export') all rows of data are fetched and then converted to the desired format. This may consume an unlimited amount of memory; when report results are particularly large, it may consume all available memory, leading to timed-out HTTP requests, crashes, and potentially data loss. (We have experienced Zebra index corruption as a result.) In master, reports/guided_reports.pl supports three export types -- tab-delimited, CSV, or *.ods. In each case, all data are loaded into memory and held there before any output is produced: 891 $sql = get_prepped_report( $sql, \@param_names, \@sql_params ); 892 my ($sth, $q_errors) = execute_query($sql); ... 895 if ($format eq 'tab') { 896 $type = 'application/octet-stream'; 897 $content .= join("\t", header_cell_values($sth)) . "\n"; 898 $content = Encode::decode('UTF-8', $content); 899 while (my $row = $sth->fetchrow_arrayref()) { 900 $content .= join("\t", @$row) . "\n"; 901 } 902 } else { 903 my $delimiter = C4::Context->preference('delimiter') || ','; 904 if ( $format eq 'csv' ) { ... 914 while (my $row = $sth->fetchrow_arrayref()) { 915 if ($csv->combine(@$row)) { 916 $content .= $csv->string() . "\n"; 917 } else { 918 push @$q_errors, { combine => $csv->error_diag() } ; 919 } 920 } 921 } 922 elsif ( $format eq 'ods' ) { ... 932 # Other line in Unicode 933 my $sql_rows = $sth->fetchall_arrayref(); 934 foreach my $sql_row ( @$sql_rows ) { 935 my @content_row; 936 foreach my $sql_cell ( @$sql_row ) { 937 push @content_row, Encode::encode( 'UTF8', $sql_cell ); 938 } 939 push @$ods_content, \@content_row; 940 } 941 942 # Process 943 generate_ods($ods_filepath, $ods_content); 944 945 # Output 946 binmode(STDOUT); 947 open $ods_fh, '<', $ods_filepath; 948 $content .= $_ while <$ods_fh>; 949 unlink $ods_filepath; 950 } The *.ods case is particularly problematic, because before any data is sent back to the user's browser, *three* copies of the full results are sitting in memory simultaneously -- @$sql_row, @$ods_content, and $content. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.