[Koha-bugs] [Bug 23685] New: Exporting report may consume unlimited memory

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23685

            Bug ID: 23685
           Summary: Exporting report may consume unlimited memory
 Change sponsored?: ---
           Product: Koha
           Version: master
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5 - low
         Component: Reports
          Assignee: koha-bugs at lists.koha-community.org
          Reporter: paul at flo.org
        QA Contact: testopia at bugs.koha-community.org

In guided_reports.pl, when exporting report results ($phase eq 'Export') all
rows of data are fetched and then converted to the desired format.  This may
consume an unlimited amount of memory; when report results are particularly
large, it may consume all available memory, leading to timed-out HTTP requests,
crashes, and potentially data loss.  (We have experienced Zebra index
corruption as a result.)

In master, reports/guided_reports.pl supports three export types --
tab-delimited, CSV, or *.ods.  In each case, all data are loaded into memory
and held there before any output is produced:

   891      $sql = get_prepped_report( $sql, \@param_names, \@sql_params );
   892          my ($sth, $q_errors) = execute_query($sql);
   ...
   895          if ($format eq 'tab') {
   896              $type = 'application/octet-stream';
   897              $content .= join("\t", header_cell_values($sth)) . "\n";
   898              $content = Encode::decode('UTF-8', $content);
   899              while (my $row = $sth->fetchrow_arrayref()) {
   900                  $content .= join("\t", @$row) . "\n";
   901              }
   902          } else {
   903              my $delimiter = C4::Context->preference('delimiter') ||
',';
   904              if ( $format eq 'csv' ) {
   ...
   914                  while (my $row = $sth->fetchrow_arrayref()) {
   915                      if ($csv->combine(@$row)) {
   916                          $content .= $csv->string() . "\n";
   917                      } else {
   918                          push @$q_errors, { combine =>
$csv->error_diag() } ;
   919                      }
   920                  }
   921              }
   922              elsif ( $format eq 'ods' ) {
   ...
   932                  # Other line in Unicode
   933                  my $sql_rows = $sth->fetchall_arrayref();
   934                  foreach my $sql_row ( @$sql_rows ) {
   935                      my @content_row;
   936                      foreach my $sql_cell ( @$sql_row ) {
   937                          push @content_row, Encode::encode( 'UTF8',
$sql_cell );
   938                      }
   939                      push @$ods_content, \@content_row;
   940                  }
   941
   942                  # Process
   943                  generate_ods($ods_filepath, $ods_content);
   944
   945                  # Output
   946                  binmode(STDOUT);
   947                  open $ods_fh, '<', $ods_filepath;
   948                  $content .= $_ while <$ods_fh>;
   949                  unlink $ods_filepath;
   950              }

The *.ods case is particularly problematic, because before any data is sent
back to the user's browser, *three* copies of the full results are sitting in
memory simultaneously -- @$sql_row, @$ods_content, and $content.

-- 
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.