[Bug 42485] New: Elasticsearch dynamic mapping date detection causes indexing failures with ARRAY MARC format
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Bug ID: 42485 Summary: Elasticsearch dynamic mapping date detection causes indexing failures with ARRAY MARC format Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: major Priority: P5 - low Component: Searching - Elasticsearch Assignee: koha-bugs@lists.koha-community.org Reporter: tomascohen@gmail.com QA Contact: testopia@bugs.koha-community.org When ElasticsearchMARCFormat is set to ARRAY, the marc_data_array field is configured with dynamic:true in field_config.yaml. Elasticsearch auto-detects field types based on the first document indexed. If a MARC subfield value looks like a date (e.g. '2024-01-15' in 952$e/booksellerid), ES permanently locks that subfield's mapping to type 'date'. All subsequent records with non-date values in the same subfield fail to index with a document_parsing_exception. This affects any subfield stored in marc_data_array that is not explicitly mapped in Koha's search field configuration â most notably item subfields like 952$e (acquisition source), 952$d (date of acquisition), and 952$w (replacement price date). The same issue applies to numeric detection: if the first value for a subfield looks numeric, ES maps it as 'long', and subsequent text values fail. **Root cause** field_config.yaml configures marc_data_array as: marc_data_array: type: object dynamic: true There is no date_detection or numeric_detection control, so ES uses its default behavior of guessing types from values. **Steps to reproduce** 1. Launch a fresh KTD: $ ktd --search-engine es8 --name broken-es --proxy up -d $ ktd --name broken-es --wait-ready 180 2. Get a shell: $ ktd --name broken-es --shell 3. Set ARRAY format, flush cache, wipe records, reset index: k$ koha-mysql kohadev -e "UPDATE systempreferences SET value='ARRAY' WHERE variable='ElasticsearchMARCFormat';" k$ echo 'flush_all' | nc -q1 memcached 11211 k$ koha-mysql kohadev -e "DELETE FROM biblio_metadata; DELETE FROM items; DELETE FROM biblioitems; DELETE FROM biblio;" k$ koha-shell kohadev -c 'rebuild_elasticsearch.pl -d -r --biblios' 4. Add a record with a date-like value in 952$e (booksellerid): k$ koha-shell kohadev -p -c 'perl -MC4::Biblio -MMARC::Record -MMARC::Field -MKoha::Item -e " my \$r = MARC::Record->new(); \$r->append_fields( MARC::Field->new(q{245}, q{1}, q{0}, a => q{Record one}), MARC::Field->new(q{942}, q{ }, q{ }, c => q{BK}), ); my (\$bn) = C4::Biblio::AddBiblio(\$r, q{}); Koha::Item->new({ biblionumber => \$bn, biblioitemnumber => \$bn, homebranch => q{CPL}, holdingbranch => q{CPL}, itype => q{BK}, booksellerid => q{2024-01-15}, })->store; print qq{Added biblio \$bn\n}; "' 5. Index it (this poisons the mapping): k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber> 6. Verify ES auto-detected 952$e as type 'date': k$ curl -s 'es:9200/koha_kohadev_biblios/_mapping/field/marc_data_array.fields.952.subfields.e' => {"type":"date"} 7. Add a second record with a non-date value in 952$e: k$ koha-shell kohadev -p -c 'perl -MC4::Biblio -MMARC::Record -MMARC::Field -MKoha::Item -e " my \$r = MARC::Record->new(); \$r->append_fields( MARC::Field->new(q{245}, q{1}, q{0}, a => q{Record two}), MARC::Field->new(q{942}, q{ }, q{ }, c => q{BK}), ); my (\$bn) = C4::Biblio::AddBiblio(\$r, q{}); Koha::Item->new({ biblionumber => \$bn, biblioitemnumber => \$bn, homebranch => q{CPL}, holdingbranch => q{CPL}, itype => q{BK}, booksellerid => q{My Vendor Name}, })->store; print qq{Added biblio \$bn\n}; "' 8. Try to index â THIS FAILS: k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber> -v -v -v => Record #N failed to parse field [marc_data_array.fields.952.subfields.e] of type [date]... 'My Vendor Name' (document_parsing_exception) **Proposed fix** Set date_detection: false and numeric_detection: false at the mapping root level in get_elasticsearch_mappings() in Koha::SearchEngine::Elasticsearch. These are index-level mapping parameters (not per-field), so they cannot be set in field_config.yaml on the marc_data_array object definition. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|koha-bugs@lists.koha-commun |tomascohen@gmail.com |ity.org | CC| |tomascohen@gmail.com -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ere.maijala@helsinki.fi, | |mark@hofstetter.at, | |nick@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |Needs Signoff Patch complexity|--- |Trivial patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 --- Comment #1 from Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> --- Created attachment 198280 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198280&action=edit Bug 42485: Disable date and numeric detection on ES mappings When ElasticsearchMARCFormat is set to ARRAY, the marc_data_array field uses dynamic:true mapping. Elasticsearch auto-detects field types based on the first document indexed. If a MARC subfield value looks like a date (e.g. 2024-01-15 in 952$e), ES locks that subfield to type date. Subsequent records with non-date values in the same subfield fail to index with a document_parsing_exception. This patch sets date_detection and numeric_detection to false at the mapping root level, so ES treats all dynamically mapped fields as text instead of guessing types. Test plan: 1. $ ktd --search-engine es8 --name test-es --proxy up -d $ ktd --name test-es --wait-ready 180 $ ktd --name test-es --shell 2. k$ koha-mysql kohadev -e "UPDATE systempreferences SET value='ARRAY' WHERE variable='ElasticsearchMARCFormat';" k$ echo 'flush_all' | nc -q1 memcached 11211 k$ koha-mysql kohadev -e "DELETE FROM biblio_metadata; DELETE FROM items; DELETE FROM biblioitems; DELETE FROM biblio;" k$ rebuild_elasticsearch.pl -d -r --biblios 3. Add a record with date-like 952$e and index it: k$ koha-shell kohadev -p -c 'perl -MC4::Biblio -MMARC::Record -MMARC::Field -MKoha::Item -e " my \$r = MARC::Record->new(); \$r->append_fields( MARC::Field->new(q{245},q{1},q{0},a=>q{Record one}), MARC::Field->new(q{942},q{ },q{ },c=>q{BK})); my (\$bn) = C4::Biblio::AddBiblio(\$r, q{}); Koha::Item->new({biblionumber=>\$bn, biblioitemnumber=>\$bn, homebranch=>q{CPL}, holdingbranch=>q{CPL}, itype=>q{BK}, booksellerid=>q{2024-01-15}})->store; print qq{Added biblio \$bn\n};"' k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber> 4. Check the mapping for 952$e: k$ curl -s es:9200/koha_kohadev_biblios/_mapping/field/\ marc_data_array.fields.952.subfields.e => FAIL: type is "date" (should be "text") 5. Add a record with non-date 952$e and index it: k$ koha-shell kohadev -p -c 'perl -MC4::Biblio -MMARC::Record -MMARC::Field -MKoha::Item -e " my \$r = MARC::Record->new(); \$r->append_fields( MARC::Field->new(q{245},q{1},q{0},a=>q{Record two}), MARC::Field->new(q{942},q{ },q{ },c=>q{BK})); my (\$bn) = C4::Biblio::AddBiblio(\$r, q{}); Koha::Item->new({biblionumber=>\$bn, biblioitemnumber=>\$bn, homebranch=>q{CPL}, holdingbranch=>q{CPL}, itype=>q{BK}, booksellerid=>q{My Vendor Name}})->store; print qq{Added biblio \$bn\n};"' k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber> -v -v -v => FAIL: document_parsing_exception - failed to parse date field [My Vendor Name] 6. Apply this patch, repeat from step 2 7. Repeat step 4: => SUCCESS: type is "text" 8. Repeat step 5: => SUCCESS: No indexing errors 9. Sign off :-D -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |januszop@gmail.com -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Tomás Cohen Arazi (tcohen) <tomascohen@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcook@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Andrew Fuerste-Henry <andrew@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198280|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 --- Comment #2 from Andrew Fuerste-Henry <andrew@bywatersolutions.com> --- Created attachment 198296 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=198296&action=edit Bug 42485: Disable date and numeric detection on ES mappings When ElasticsearchMARCFormat is set to ARRAY, the marc_data_array field uses dynamic:true mapping. Elasticsearch auto-detects field types based on the first document indexed. If a MARC subfield value looks like a date (e.g. 2024-01-15 in 952$e), ES locks that subfield to type date. Subsequent records with non-date values in the same subfield fail to index with a document_parsing_exception. This patch sets date_detection and numeric_detection to false at the mapping root level, so ES treats all dynamically mapped fields as text instead of guessing types. Test plan: 1. $ ktd --search-engine es8 --name test-es --proxy up -d $ ktd --name test-es --wait-ready 180 $ ktd --name test-es --shell 2. k$ koha-mysql kohadev -e "UPDATE systempreferences SET value='ARRAY' WHERE variable='ElasticsearchMARCFormat';" k$ echo 'flush_all' | nc -q1 memcached 11211 k$ koha-mysql kohadev -e "DELETE FROM biblio_metadata; DELETE FROM items; DELETE FROM biblioitems; DELETE FROM biblio;" k$ rebuild_elasticsearch.pl -d -r --biblios 3. Add a record with date-like 952$e and index it: k$ koha-shell kohadev -p -c 'perl -MC4::Biblio -MMARC::Record -MMARC::Field -MKoha::Item -e " my \$r = MARC::Record->new(); \$r->append_fields( MARC::Field->new(q{245},q{1},q{0},a=>q{Record one}), MARC::Field->new(q{942},q{ },q{ },c=>q{BK})); my (\$bn) = C4::Biblio::AddBiblio(\$r, q{}); Koha::Item->new({biblionumber=>\$bn, biblioitemnumber=>\$bn, homebranch=>q{CPL}, holdingbranch=>q{CPL}, itype=>q{BK}, booksellerid=>q{2024-01-15}})->store; print qq{Added biblio \$bn\n};"' k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber> 4. Check the mapping for 952$e: k$ curl -s es:9200/koha_kohadev_biblios/_mapping/field/\ marc_data_array.fields.952.subfields.e => FAIL: type is "date" (should be "text") 5. Add a record with non-date 952$e and index it: k$ koha-shell kohadev -p -c 'perl -MC4::Biblio -MMARC::Record -MMARC::Field -MKoha::Item -e " my \$r = MARC::Record->new(); \$r->append_fields( MARC::Field->new(q{245},q{1},q{0},a=>q{Record two}), MARC::Field->new(q{942},q{ },q{ },c=>q{BK})); my (\$bn) = C4::Biblio::AddBiblio(\$r, q{}); Koha::Item->new({biblionumber=>\$bn, biblioitemnumber=>\$bn, homebranch=>q{CPL}, holdingbranch=>q{CPL}, itype=>q{BK}, booksellerid=>q{My Vendor Name}})->store; print qq{Added biblio \$bn\n};"' k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber> -v -v -v => FAIL: document_parsing_exception - failed to parse date field [My Vendor Name] 6. Apply this patch, repeat from step 2 7. Repeat step 4: => SUCCESS: type is "text" 8. Repeat step 5: => SUCCESS: No indexing errors 9. Sign off :-D Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Signed Off |Passed QA Sponsorship status|--- |Unsponsored QA Contact|testopia@bugs.koha-communit |martin.renvoize@openfifth.c |y.org |o.uk Patch complexity|Trivial patch |Small patch -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #198296|0 |1 is obsolete| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 --- Comment #3 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199926 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199926&action=edit Bug 42485: Disable date and numeric detection on ES mappings When ElasticsearchMARCFormat is set to ARRAY, the marc_data_array field uses dynamic:true mapping. Elasticsearch auto-detects field types based on the first document indexed. If a MARC subfield value looks like a date (e.g. 2024-01-15 in 952$e), ES locks that subfield to type date. Subsequent records with non-date values in the same subfield fail to index with a document_parsing_exception. This patch sets date_detection and numeric_detection to false at the mapping root level, so ES treats all dynamically mapped fields as text instead of guessing types. Test plan: 1. $ ktd --search-engine es8 --name test-es --proxy up -d $ ktd --name test-es --wait-ready 180 $ ktd --name test-es --shell 2. k$ koha-mysql kohadev -e "UPDATE systempreferences SET value='ARRAY' WHERE variable='ElasticsearchMARCFormat';" k$ echo 'flush_all' | nc -q1 memcached 11211 k$ koha-mysql kohadev -e "DELETE FROM biblio_metadata; DELETE FROM items; DELETE FROM biblioitems; DELETE FROM biblio;" k$ rebuild_elasticsearch.pl -d -r --biblios 3. Add a record with date-like 952$e and index it: k$ koha-shell kohadev -p -c 'perl -MC4::Biblio -MMARC::Record -MMARC::Field -MKoha::Item -e " my \$r = MARC::Record->new(); \$r->append_fields( MARC::Field->new(q{245},q{1},q{0},a=>q{Record one}), MARC::Field->new(q{942},q{ },q{ },c=>q{BK})); my (\$bn) = C4::Biblio::AddBiblio(\$r, q{}); Koha::Item->new({biblionumber=>\$bn, biblioitemnumber=>\$bn, homebranch=>q{CPL}, holdingbranch=>q{CPL}, itype=>q{BK}, booksellerid=>q{2024-01-15}})->store; print qq{Added biblio \$bn\n};"' k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber> 4. Check the mapping for 952$e: k$ curl -s es:9200/koha_kohadev_biblios/_mapping/field/\ marc_data_array.fields.952.subfields.e => FAIL: type is "date" (should be "text") 5. Add a record with non-date 952$e and index it: k$ koha-shell kohadev -p -c 'perl -MC4::Biblio -MMARC::Record -MMARC::Field -MKoha::Item -e " my \$r = MARC::Record->new(); \$r->append_fields( MARC::Field->new(q{245},q{1},q{0},a=>q{Record two}), MARC::Field->new(q{942},q{ },q{ },c=>q{BK})); my (\$bn) = C4::Biblio::AddBiblio(\$r, q{}); Koha::Item->new({biblionumber=>\$bn, biblioitemnumber=>\$bn, homebranch=>q{CPL}, holdingbranch=>q{CPL}, itype=>q{BK}, booksellerid=>q{My Vendor Name}})->store; print qq{Added biblio \$bn\n};"' k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber> -v -v -v => FAIL: document_parsing_exception - failed to parse date field [My Vendor Name] 6. Apply this patch, repeat from step 2 7. Repeat step 4: => SUCCESS: type is "text" 8. Repeat step 5: => SUCCESS: No indexing errors 9. Sign off :-D Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 --- Comment #4 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 199927 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=199927&action=edit Bug 42485: (QA follow-up) Add explicit JSON::PP import and test coverage Add explicit `use JSON::PP ()` to Elasticsearch.pm and the test file to make the dependency on JSON::PP::false visible rather than relying on YAML::XS loading it as a side effect. Extend the get_elasticsearch_mappings() test subtest to assert that date_detection and numeric_detection are disabled at the mapping root. Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Passed QA |Pushed to main Version(s)| |26.11.00 released in| | -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 --- Comment #5 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Thanks everyone! Pushed to main for 26.11! -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 --- Comment #6 from Pedro Amorim (ammopt) <pedro.amorim@openfifth.co.uk> --- Created attachment 200423 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=200423&action=edit Bug 42485: (RM follow-up): Tidy prove xt/perltidy.t Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Lucas Gass (lukeg) <lucas@bywatersolutions.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Version(s)|26.11.00 |26.11.00,26.05.01 released in| | Status|Pushed to main |Pushed to stable -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 --- Comment #7 from Lucas Gass (lukeg) <lucas@bywatersolutions.com> --- Nice work everyone! Pushed to 26.05.x for the upcoming 26.05.01 release. -- You are receiving this mail because: You are watching all bug changes.
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42485 Baptiste Wojtkowski (bwoj) <baptiste.wojtkowski@biblibre.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Pushed to stable |Pushed to oldstable CC| |baptiste.wojtkowski@biblibr | |e.com Version(s)|26.11.00,26.05.01 |26.11.00,26.05.01,25.11.07 released in| | --- Comment #8 from Baptiste Wojtkowski (bwoj) <baptiste.wojtkowski@biblibre.com> --- Pushed into 25.11.x for 25.11.07 nice work everyone -- You are receiving this mail because: You are watching all bug changes.
participants (1)
-
bugzilla-daemon＠bugs.koha-community.org