https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=43584 Bug ID: 43584 Summary: Elasticsearch mapping changes that require a full reindex silently leave the live index broken until an admin manually reindexes Initiative type: --- Sponsorship --- status: Product: Koha Version: Main Hardware: All OS: All Status: NEW Severity: normal Priority: P5 - low Component: Searching Assignee: koha-bugs@lists.koha-community.org Reporter: martin.renvoize@openfifth.co.uk QA Contact: testopia@bugs.koha-community.org Target Milestone: --- When a library admin changes a search field's mapping in Administration > Search engine configuration (Elasticsearch) (admin/searchengine/elasticsearch/mappings.pl) in a way that isn't purely additive (e.g. changing a field's type), Koha does not reindex or roll back - it leaves the live Elasticsearch index in a broken, mismatched state indefinitely, with no visible warning to end users and no automatic recovery. The only fix is a library admin/sysadmin noticing the problem and manually running a full misc/search_tools/rebuild_elasticsearch.pl -d (or -r), which for a large catalogue can mean a long window of degraded/incorrect search results. Root cause (traced in Koha/SearchEngine/Elasticsearch/Indexer.pm and admin/searchengine/elasticsearch/mappings.pl): 1. Saving the mappings form immediately calls Indexer->update_mappings(), which issues a live Elasticsearch PUT _mapping against the existing index (Indexer.pm update_mappings, around line 262). 2. Elasticsearch's put_mapping API can only add genuinely new fields - it cannot change an already-mapped field's type/analyzer. When the change isn't purely additive, put_mapping throws. Koha catches this, sets the index status to "recreate required" (INDEX_STATUS_RECREATE_REQUIRED), and shows an error banner - but only on the mappings.pl admin screen itself. 3. That index status is never checked anywhere else in the codebase - not in search.pl, not in the OPAC, not in the live cataloguing indexing pipeline (Koha::SearchEngine::Elasticsearch::Indexer::update_index, called on every biblio/item save). So cataloguing and searching both continue to hit the now-inconsistent index with no gating or warning. 4. Nothing schedules or forces the required full reindex. It's entirely a manual, unscheduled, blocking operation the admin must remember to run. I reproduced this on main (2026-09-21, es8) as follows: Test plan / reproduction steps: 1. Set up KTD with Elasticsearch (e.g. ktd --search-engine es8 up -d), set the SearchEngine system preference to "Elasticsearch", and run a full reindex (misc/search_tools/rebuild_elasticsearch.pl -r -v) so the index and DB mapping config agree. 2. Confirm a normal search works, e.g. search for a known title in the OPAC or staff interface. 3. Go to Administration > Search engine configuration (Elasticsearch) > Bibliographic records. 4. Edit an existing, already-mapped field that has real indexed data - for example change "title" from type "String" to type "Number" - and save. 5. Observe: the page reports an error and Koha logs (or, reproduced directly in Perl): Unable to update mappings for index "koha_<instance>_biblios". Reason was: "mapper [title] cannot be changed from type [text] to [integer]". Index needs to be recreated and reindexed 6. Note that this error is only visible on the mappings.pl page itself. Go to the OPAC or staff client and search - searches continue to run against the still-mismatched index with no warning to the person searching, and results/behaviour for the affected field are now undefined/inconsistent with the field's newly configured type and options (facets, sort, filters depending on the field silently misbehave). 7. The only way to recover is for someone to notice the admin banner and run, e.g.: misc/search_tools/rebuild_elasticsearch.pl -d -b -v (drop & recreate + reindex biblios), which for a large catalogue can take a long time, during which search functionality for the affected field(s) stays broken. Note this isn't limited to "hard failure" cases like the type-change example above. Even a purely additive mapping change (e.g. adding a new facet on an existing field) succeeds immediately at the Elasticsearch level, but every record indexed before the change has no value in the new field - so facets/filters/sort on it are silently incomplete for the whole existing catalogue until a full reindex populates it. There is no indication to a searching user (or even the admin) that results are incomplete. Suggested directions (for discussion, not yet implemented): A. Blue/green reindex via index aliases: point the Elasticsearch index name Koha reads/writes as an alias (rather than the literal <instance>_biblios index it uses today - see Koha::SearchEngine::Elasticsearch::Indexer create_index/update_mappings/index_name) at a concrete versioned index (<instance>_biblios_v3). When a mapping change requires a rebuild, build a new concrete index with the new mappings in the background, reindex into it, then atomically flip the alias once it's caught up (capturing any records changed during the rebuild window in a final catch-up pass), and drop the old index. This is the standard Elasticsearch zero-downtime reindex pattern and would mean searches keep working against the old, still-consistent index throughout, with no admin-visible downtime at all. B. Scheduled off-hours full reindex with fallback: rather than applying an incompatible mapping change immediately, queue it, keep serving searches against the current (old-mapping) index/config, and run the full reindex as a background/cron job during a configured off-hours window, then apply the new mapping config atomically once the reindex completes. B is strictly weaker than A (still involves a window where either old or new config must be chosen, and doesn't help instances that want the change live sooner), so A is the preferred direction, but is a bigger architectural change (introducing alias-based indices, a background full-reindex job, and a "reindex in progress" status) than B. Either would be a large enough change to warrant an RFC / community discussion before a patch is written - filing this now primarily to document the problem, root cause, and reproduction steps, and to open discussion on which direction (or another one entirely) the community prefers. At minimum, independent of A/B, two smaller, cheap wins are worth considering regardless of the larger direction chosen: - Surface the "recreate required" / "reindex required" index status somewhere visible beyond the mappings.pl page (e.g. a staff client system alert), so admins aren't only warned if they happen to revisit that specific screen. - Have mappings.pl warn before saving when a proposed type change is likely to be non-additive (best-effort - comparing old vs new type), rather than only after the fact. -- You are receiving this mail because: You are watching all bug changes. You are the assignee for the bug.