https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Bug ID: 42485 Summary: Elasticsearch dynamic mapping date detection causes indexing failures with ARRAY MARC format Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: tomascohen@gmail.com QA Contact: testopia@bugs.koha-community.org When ElasticsearchMARCFormat is set to ARRAY, the marc_data_array field is configured with dynamic:true in field_config.yaml. Elasticsearch auto-detects field types based on the first document indexed. If a MARC subfield value looks like a date (e.g. '2024-01-15' in 952$e/booksellerid), ES permanently locks that subfield's mapping to type 'date'. All subsequent records with non-date values in the same subfield fail to index with a document_parsing_exception. This affects any subfield stored in marc_data_array that is not explicitly mapped in Koha's search field configuration â most notably item subfields like 952$e (acquisition source), 952$d (date of acquisition), and 952$w (replacement price date). The same issue applies to numeric detection: if the first value for a subfield looks numeric, ES maps it as 'long', and subsequent text values fail. **Root cause** field_config.yaml configures marc_data_array as: marc_data_array: type: object dynamic: true There is no date_detection or numeric_detection control, so ES uses its default behavior of guessing types from values. **Steps to reproduce** 1. Launch a fresh KTD: $ ktd --search-engine es8 --name broken-es --proxy up -d $ ktd --name broken-es --wait-ready 180 2. Get a shell: $ ktd --name broken-es --shell 3. Set ARRAY format, flush cache, wipe records, reset index: k$ koha-mysql kohadev -e "UPDATE systempreferences SET value='ARRAY' WHERE variable='ElasticsearchMARCFormat';" k$ echo 'flush_all' | nc -q1 memcached 11211 k$ koha-mysql kohadev -e "DELETE FROM biblio_metadata; DELETE FROM items; DELETE FROM biblioitems; DELETE FROM biblio;" k$ koha-shell kohadev -c 'rebuild_elasticsearch.pl -d -r --biblios' 4. Add a record with a date-like value in 952$e (booksellerid): k$ koha-shell kohadev -p -c 'perl -MC4::Biblio -MMARC::Record -MMARC::Field -MKoha::Item -e " my \$r = MARC::Record->new(); \$r->append_fields( MARC::Field->new(q{245}, q{1}, q{0}, a => q{Record one}), MARC::Field->new(q{942}, q{ }, q{ }, c => q{BK}), ); my (\$bn) = C4::Biblio::AddBiblio(\$r, q{}); Koha::Item->new({ biblionumber => \$bn, biblioitemnumber => \$bn, homebranch => q{CPL}, holdingbranch => q{CPL}, itype => q{BK}, booksellerid => q{2024-01-15}, })->store; print qq{Added biblio \$bn\n}; "' 5. Index it (this poisons the mapping): k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber> 6. Verify ES auto-detected 952$e as type 'date': k$ curl -s 'es:9200/koha_kohadev_biblios/_mapping/field/marc_data_array.fields.952.subfields.e' => {"type":"date"} 7. Add a second record with a non-date value in 952$e: k$ koha-shell kohadev -p -c 'perl -MC4::Biblio -MMARC::Record -MMARC::Field -MKoha::Item -e " my \$r = MARC::Record->new(); \$r->append_fields( MARC::Field->new(q{245}, q{1}, q{0}, a => q{Record two}), MARC::Field->new(q{942}, q{ }, q{ }, c => q{BK}), ); my (\$bn) = C4::Biblio::AddBiblio(\$r, q{}); Koha::Item->new({ biblionumber => \$bn, biblioitemnumber => \$bn, homebranch => q{CPL}, holdingbranch => q{CPL}, itype => q{BK}, booksellerid => q{My Vendor Name}, })->store; print qq{Added biblio \$bn\n}; "' 8. Try to index â THIS FAILS: k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber> -v -v -v => Record #N failed to parse field [marc_data_array.fields.952.subfields.e] of type [date]... 'My Vendor Name' (document_parsing_exception) **Proposed fix** Set date_detection: false and numeric_detection: false at the mapping root level in get_elasticsearch_mappings() in Koha::SearchEngine::Elasticsearch. These are index-level mapping parameters (not per-field), so they cannot be set in field_config.yaml on the marc_data_array object definition. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.