https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39537 Bug ID: 39537 Summary: bulkmarcimport.pl fails to import large files Change sponsored?: --- Product: Koha Version: unspecified Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Command-line Utilities Assignee: koha-bugs@lists.koha-community.org Reporter: januszop@gmail.com QA Contact: testopia@bugs.koha-community.org CC: glasklas@gmail.com, martin.renvoize@openfifth.co.uk, robin@catalyst.net.nz, schodkowy.omegi-0r@icloud.com After applying bug 29440 (entered in 24.05), bulkmarcimport.pl fails to import large files. The reason is that, after the changes made there, there are now two loops iterating the records to be imported, first of which reads the file and places the read records in an Perl array in memory. The second loop takes the records from that array array and actually inserts them into Koha database. Now, imagine what happens with limited RAM on the system and a large file (100K+, 1M+, 5M+ records, depending on system configuration) -- the script consumes more and more memory and at the end destroys the system stability, possibly being killed because of oom. I must say, bulkmarcimport.pl has a lot of useful features, but has become completely useless for larger datasets and one have now to write own simple scripts for each use case to cover actual needs and adapting elements from the original script. This is tiring. I am wondering what was the reason of such a design (splitting the main loop and collecting the records in a growing array in memory). Should it not be corrected? I would be happy to know David's (the original author's) opinion. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.