https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 --- Comment #1 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 176698 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=176698&action=edit Bug 38913: (bug 38416 follow-up) Elasticsearch indexing explodes with oversized records After Bug 38416 Elasticsearch indexing explodes with oversized records, especially with UTF encoded data. In Koha::SearchEngine::Elasticsearch::marc_records_to_documents a following snippet has been introduced: my $usmarc_record = $record->as_usmarc(); my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); But if $record is oversized (> 99999 bytes), it is OK for MARC::Record object, but not for $record->as_usmarc. The produced ISO 2709 string is not correct and hence cannot be properly converted back to MARC::Record object by new_from_usmarc. The result in this case can be like: UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. Since it is done without any eval / try, the whole reindex procedure (for instance rebuild_elasticsearch.pl) is being randomly interrupted with no explanation. Test plan: ========== Hard to reproduce. But the explanation together with discussion in Bug 38416 (from 2024-12-15) explains and justifies the need of this added eval. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au, | |m.de.rooy@rijksmuseum.nl, | |martin.renvoize@ptfs-europe | |.com, | |nick@bywatersolutions.com, | |nugged@gmail.com -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 Janusz Kaczmarek <januszop@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #176698|0 |1 is obsolete| | --- Comment #3 from Janusz Kaczmarek <januszop@gmail.com> --- Created attachment 176701 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=176701&action=edit Bug 38913: (bug 38416 follow-up) Elasticsearch indexing explodes with oversized records After Bug 38416 Elasticsearch indexing explodes with oversized records, especially with UTF encoded data. In Koha::SearchEngine::Elasticsearch::marc_records_to_documents a following snippet has been introduced: my $usmarc_record = $record->as_usmarc(); my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); But if $record is oversized (> 99999 bytes), it is OK for MARC::Record object, but not for $record->as_usmarc. The produced ISO 2709 string is not correct and hence cannot be properly converted back to MARC::Record object by new_from_usmarc. The result in this case can be like: UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. Since it is done without any eval / try, the whole reindex procedure (for instance rebuild_elasticsearch.pl) is being randomly interrupted with no explanation. Test plan: ========== Hard to reproduce. But the explanation together with discussion in Bug 38416 (from 2024-12-15) explains and justifies the need of this added eval. 1. Have a standard KTD installation with Elasticsearch. 2. Use the provided test record - add it to Koha with ./misc/migration_tools/bulkmarcimport.pl -b -file test.xml -m=MARCXML (have patience). During load process you should see a message like: UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. 3. The record should get biblionumber 439. Check in librarian interface with http://<your_addreess>:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=439 that the record has been imported. However, you should not be able to make a search for this record. 4. Try to reindex with: ./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439 You should get a message like: UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. Again, no search results. 5. Apply the patch ; restart_all. 6. Repeat reindex with: ./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439 There should be no warning now and you should be able to find the record. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 Magnus Enger <magnus@libriotech.no> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #176701|0 |1 is obsolete| | --- Comment #4 from Magnus Enger <magnus@libriotech.no> --- Created attachment 176704 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=176704&action=edit Bug 38913: (bug 38416 follow-up) Elasticsearch indexing explodes with oversized records After Bug 38416 Elasticsearch indexing explodes with oversized records, especially with UTF encoded data. In Koha::SearchEngine::Elasticsearch::marc_records_to_documents a following snippet has been introduced: my $usmarc_record = $record->as_usmarc(); my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); But if $record is oversized (> 99999 bytes), it is OK for MARC::Record object, but not for $record->as_usmarc. The produced ISO 2709 string is not correct and hence cannot be properly converted back to MARC::Record object by new_from_usmarc. The result in this case can be like: UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. Since it is done without any eval / try, the whole reindex procedure (for instance rebuild_elasticsearch.pl) is being randomly interrupted with no explanation. Test plan: ========== Hard to reproduce. But the explanation together with discussion in Bug 38416 (from 2024-12-15) explains and justifies the need of this added eval. 1. Have a standard KTD installation with Elasticsearch. 2. Use the provided test record - add it to Koha with ./misc/migration_tools/bulkmarcimport.pl -b -file test.xml -m=MARCXML (have patience). During load process you should see a message like: UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. 3. The record should get biblionumber 439. Check in librarian interface with http://<your_addreess>:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=439 that the record has been imported. However, you should not be able to make a search for this record. 4. Try to reindex with: ./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439 You should get a message like: UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. Again, no search results. 5. Apply the patch ; restart_all. 6. Repeat reindex with: ./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439 There should be no warning now and you should be able to find the record. Signed-off-by: Magnus Enger <magnus@libriotech.no> Followed the test plan. Works as advertised. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 --- Comment #6 from David Cook <dcook@prosentient.com.au> --- Guessing I must need to update my koha-testing-docker because it's failing with a missing MARC/Lint.pm module... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #176704|0 |1 is obsolete| | --- Comment #8 from David Cook <dcook@prosentient.com.au> --- Created attachment 176767 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=176767&action=edit Bug 38913: (bug 38416 follow-up) Elasticsearch indexing explodes with oversized records After Bug 38416 Elasticsearch indexing explodes with oversized records, especially with UTF encoded data. In Koha::SearchEngine::Elasticsearch::marc_records_to_documents a following snippet has been introduced: my $usmarc_record = $record->as_usmarc(); my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); But if $record is oversized (> 99999 bytes), it is OK for MARC::Record object, but not for $record->as_usmarc. The produced ISO 2709 string is not correct and hence cannot be properly converted back to MARC::Record object by new_from_usmarc. The result in this case can be like: UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. Since it is done without any eval / try, the whole reindex procedure (for instance rebuild_elasticsearch.pl) is being randomly interrupted with no explanation. Test plan: ========== Hard to reproduce. But the explanation together with discussion in Bug 38416 (from 2024-12-15) explains and justifies the need of this added eval. 1. Have a standard KTD installation with Elasticsearch. 2. Use the provided test record - add it to Koha with ./misc/migration_tools/bulkmarcimport.pl -b -file test.xml -m=MARCXML (have patience). During load process you should see a message like: UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. 3. The record should get biblionumber 439. Check in librarian interface with http://<your_addreess>:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=439 that the record has been imported. However, you should not be able to make a search for this record. 4. Try to reindex with: ./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439 You should get a message like: UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. Again, no search results. 5. Apply the patch ; restart_all. 6. Repeat reindex with: ./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439 There should be no warning now and you should be able to find the record. Signed-off-by: Magnus Enger <magnus@libriotech.no> Followed the test plan. Works as advertised. Signed-off-by: David Cook <dcook@prosentient.com.au> -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 --- Comment #9 from David Cook <dcook@prosentient.com.au> --- Something is still bugging me... t/db_dependent/Koha/SearchEngine/Elasticsearch.t In theory, that test script has a test for a "large MARC record" which runs marc_records_to_documents() and it doesn't fail on "main". It must be longer than 99999 bytes since it's switching to MARCXML from base64ISO2709. So we did test for large MARC records om bug 38416. But... since the fatal error Janusz is fixing comes from MARC::File::USMARC not handling an exception during marc_to_utf8(), it must also be because being over 99999 bytes creates an invalid USMARC directory in the USMARC data. It then starts doing string handling using the positional math. And then it needs to get the right combination of invalid bytes. Since the unit test was just using single byte ASCII... it should be impossible to generate invalid UTF8 in this particular test scenario. Let's see if I can break the unit test in main... -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 --- Comment #11 from David Cook <dcook@prosentient.com.au> --- Created attachment 176768 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=176768&action=edit Bug 38913: (QA follow-up) test UTF-8 exceptions in large MARC records MARC records with over 99999 bytes are invalid by spec, and when you use UTF-8 encoded characters in your MARC records, there is the potential to generate fatal errors in MARC::File::USMARC when it runs "marc_to_utf8" from "MARC::File::Encode" during its "decode" operation. That is, if you MARC::File::USMARC->encode a MARC record with over 99999 bytes (including a number of UTF-8 bytes), there is the potential when you run MARC::File:USMARC->decode on that same data that you'll generate a fatal exception. The main patch in bug 38913 wraps the function doing the decode, so that a bad record doesn't crash processing. Without the patch, this unit test will fail. With the patch, this unit test will pass. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 Jonathan Druart <jonathan.druart@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jonathan.druart@gmail.com --- Comment #12 from Jonathan Druart <jonathan.druart@gmail.com> --- This is clearly not enough (could go on a separate bugs). Testing this patch I have noticed several things: 1. $ ./misc/migration_tools/bulkmarcimport.pl -b -file test.xml -m=MARCXML .UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. Not really useful to guess where the error is, but we know it's in the file so we can search in it easily 2. Stage the file and import using the UI id: 2 status: failed progress: 0 size: 1 borrowernumber: 51 type: marc_import_commit_batch queue: long_tasks data: {"report":{"import_batch_id":"1","num_items_added":null,"num_ignored":null,"num_items_replaced":null,"num_items_errored":null,"num_added":null,"num_updated":null},"import_batch_id":"1","overlay_fra mework":null,"messages":[],"frameworkcode":""} context: {"firstname":null,"surname":"koha","flags":"1","number":"51","register_name":null,"emailaddress":null,"desk_id":null,"desk_name":null,"branchname":"Centerville","register_id":null,"shibboleth":"0", "cardnumber":"42","branch":"CPL","interface":"intranet","id":"koha"} enqueued_on: 2025-01-20 08:40:28 started_on: 2025-01-20 08:40:29 ended_on: 2025-01-20 08:40:29 No info on the failure. There is something in the log, but that should go in the report of the job ==> /var/log/koha/kohadev/worker-output.log <== Record length of 527856 is larger than the MARC spec allows (99999 bytes). at /usr/share/perl5/MARC/File/USMARC.pm line 314. UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. 3. Now the record is in the DB, start a full reindex: % koha-elasticsearch --rebuild -b kohadev UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. Something went wrong rebuilding indexes for kohadev No info on the problematic record! We should tell which record failed. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 --- Comment #15 from Janusz Kaczmarek <januszop@gmail.com> --- I've created Bug 38933 for this stage/import from UI issue. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 --- Comment #17 from David Cook <dcook@prosentient.com.au> --- (In reply to Jonathan Druart from comment #12)

This is clearly not enough (could go on a separate bugs).

Yep. Step by step. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 --- Comment #19 from David Cook <dcook@prosentient.com.au> --- (In reply to Janusz Kaczmarek from comment #18)

(In reply to David Cook from comment #16)

...

I raised Bug 32638 a couple years ago. I'm sure there's a bunch of reports about the MARC import failing silently.

At first glance, this seems to be a different (but somehow related) problem. The cause of 32638 seems to lie elsewhere, not in the MARC transformation itself. Am I right?

Yeah, I just meant that the MARC import doesn't surface errors/failures. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 --- Comment #21 from Martin Renvoize (ashimema) <martin.renvoize@ptfs-europe.com> --- This all reminds me a little about: Bug 35104 - We should warn when attempting to save MARC records that contain characters invalid in XML Whilst it's not specifically about record length, it's meant to try and prevent bad data making it's way into Koha entirely. That said, it sounds like this isn't "bad" data so much as just data our MARC utilities don't deal with well. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 --- Comment #23 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to Janusz Kaczmarek from comment #22)

(In reply to Jonathan Druart from comment #20)

...

...

Well, every Koha::Item->store triggers $indexer->index_records, so no wonder -- we have 2508 952 fields in the test record :)

Yes, the "interesting" was sarcastic, hence the "..." but that was not obvious, sorry.

It's still a bug IMO. Especially with this: 718 $indexer->update_index( \@search_engine_record_ids, \@search_engine_records ) unless $skip_indexing;

Does it mean that both in bulkmarcimport and in import staged records from UI we should add bibliographic records with { skip_record_index => 1 } and then add items with { skip_record_index => 1 }, and then, at the very end, or after a certain number of records, or after each record, explicitly call:

$indexer->index_records( $biblionumber(s), ...) ?

Would it be a right way?

Yes, see what we do in Koha::Items->batch_update. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 --- Comment #25 from Jonathan Druart <jonathan.druart@gmail.com> --- (In reply to Katrin Fischer from comment #24)

I see there is still a lot of discussion gong on - is it ok to push these patches as is and continue on another bug for remaining issues or should I wait?

You can push. -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 Katrin Fischer <katrin.fischer@bsz-bw.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |rel_24_11_candidate Version|unspecified |Main -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 --- Comment #26 from Katrin Fischer <katrin.fischer@bsz-bw.de> --- Pushed for 25.05! Well done everyone, thank you! -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 Paul Derscheid <paul.derscheid@lmscloud.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|25.05.00 |25.05.00,24.11.02 released in| | Status|Pushed to main |Pushed to stable -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 Paul Derscheid <paul.derscheid@lmscloud.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |paul.derscheid@lmscloud.de Keywords|rel_24_11_candidate | -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 David Cook <dcook@prosentient.com.au> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocks|39104 | See Also| |https://bugs.koha-community | |.org/bugzilla3/show_bug.cgi | |?id=39104 Referenced Bugs: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39104 [Bug 39104] Elasticsearch indexing crashes with exception in catch block -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 Alex Buckley <alexbuckley@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|25.05.00,24.11.02 |25.05.00,24.11.02,24.05.07 released in| | Status|Pushed to stable |Pushed to oldstable CC| |alexbuckley@catalyst.net.nz --- Comment #29 from Alex Buckley <alexbuckley@catalyst.net.nz> --- Backported to 24.05.x for 24.05.07 Note: The test plan of the first patch worked exactly as expected. We ran the t/db_dependent/Koha/SearchEngine/Elasticsearch. unit test before and after applying the second patch and it was successful both times. Please let us know if we should not backport the second patch to 24.05 -- You are receiving this mail because: You are watching all bug changes.

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913 Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|Pushed to oldstable |RESOLVED CC| |wainuiwitikapark@catalyst.n | |et.nz --- Comment #31 from Wainui Witika-Park <wainuiwitikapark@catalyst.net.nz> --- Not backporting to 22.11 unless requested -- You are receiving this mail because: You are watching all bug changes.