[Koha-bugs] [Bug 23766] New: Allow MARC record importer to handle large batches without causing memory issues

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23766

            Bug ID: 23766
           Summary: Allow MARC record importer to handle large batches
                    without causing memory issues
 Change sponsored?: ---
           Product: Koha
           Version: unspecified
          Hardware: All
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5 - low
         Component: Architecture, internals, and plumbing
          Assignee: koha-bugs at lists.koha-community.org
          Reporter: kyle at bywatersolutions.com
        QA Contact: testopia at bugs.koha-community.org

Right now, no matter how large a marc batch file is, the marc record stager
loads all the records into memory ( see
http://git.koha-community.org/gitweb/?p=koha.git;a=blob;f=C4/ImportBatch.pm;h=8d39ed20347132e34ab5661dedd15548df672708;hb=HEAD#l1545
).

We should break this up and process chunks of records at a time to keep the
memory footprint low for servers without enough RAM to handle this memory load.

My proposal is to modify process_batch in stage_file to loop over the records
processing X number of records at a time. X should be configurable via a
syspref.

In addition, this will require modifying C4::ImporBatch::RecordsFromMARCXMLFile
to accept an offset and a count such that it will start at the offset and
return the given count of records.

-- 
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.