https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 --- Comment #14 from Janusz Kaczmarek <januszop@gmail.com> --- (In reply to Jonathan Druart from comment #13)
With this patch: "1 MARC records done in 81.9053399562836 seconds"
However, I have delete all biblio and background_jobs before the import and now I have:
MariaDB [koha_kohadev]> select count(*) from biblio\G count(*): 1
MariaDB [koha_kohadev]> select count(*) from background_jobs\G count(*): 2508
Interesting!...
Well, every Koha::Item->store triggers $indexer->index_records, so no wonder -- we have 2508 952 fields in the test record :)
2. Stage the file and import using the UI
Same with this patch.
This is also more or less clear. In C4::ImportBatch::BatchCommitRecords called by the worker we call: my $marc_record = MARC::Record->new_from_usmarc($rowref->{'marc'}); despite of having also the marcxml representation of the record in the import_records table (import_records.marcxml). This is exactly the same issue that made David's patch die with this kind of records. Worker dies because if the uncaught die generated by new_from_usmarc. This has nothing to do with the patch (and with the previous David's patch) -- just another case of a call to a function that potentially dies without any eval / try. Now, if we create and save to import_records table both versions (iso2709 and marcxml) in C4::ImportBatch::_create_import_record, why not to use marcxml version in C4::ImportBatch::BatchCommitRecords instead of iso2709 which creates trouble in case of oversized records? After this little change it seems to work - I was able to import the huge test record with UI: diff --git a/C4/ImportBatch.pm b/C4/ImportBatch.pm index 5aebaafacf..799b69f0ca 100644 --- a/C4/ImportBatch.pm +++ b/C4/ImportBatch.pm @@ -531,7 +531,7 @@ sub BatchCommitRecords { my $item_tag; my $item_subfield; my $dbh = C4::Context->dbh; - my $sth = $dbh->prepare("SELECT import_records.import_record_id, record_type, status, overlay_status, marc, encoding + my $sth = $dbh->prepare("SELECT import_records.import_record_id, record_type, status, overlay_status, marc, marcxml, encoding FROM import_records LEFT JOIN import_auths ON (import_records.import_record_id=import_auths.import_record_id) LEFT JOIN import_biblios ON (import_records.import_record_id=import_biblios.import_record_id) @@ -568,7 +568,7 @@ sub BatchCommitRecords { } else { $marc_type = 'USMARC'; } - my $marc_record = MARC::Record->new_from_usmarc($rowref->{'marc'}); + my $marc_record = MARC::Record->new_from_xml($rowref->{'marcxml'}, $rowref->{'encoding'}); if ($record_type eq 'biblio') { # remove any item tags - rely on _batchCommitItems
3. Now the record is in the DB, start a full reindex:
% koha-elasticsearch --rebuild -b kohadev UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. Something went wrong rebuilding indexes for kohadev
No info on the problematic record! We should tell which record failed.
We don't have anything in the output, which is problematic IMO.
Yes, this is problematic, because new_from_usmarc died and we didn't catch it. But now since we call it in eval we should be save with this. -- You are receiving this mail because: You are watching all bug changes.