https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42880 --- Comment #6 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 208524 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=208524&action=edit Benchmark scripts for full-reindex / batch indexing (bug 42881 family) Clarifying the goal of this work (bug 42881 family): the target is infrastructure-facing - the time and CPU a FULL REINDEX (misc/search_tools/rebuild_elasticsearch.pl) and LARGE BATCH re-indexes (batch record modification, imports; these enqueue index jobs of up to chunk_size = 5000 records) consume. It is not about how quickly a cataloguer sees one edit: in KTD we measured the edit-to-searchable time through the stock es_indexer_daemon and it is dominated by job dispatch (roughly one job per second), so these patches do not change it. I am attaching a small benchmark package (bug42881-es-indexing-bench.tar.gz, see README.txt inside) so testers can reproduce the before/after numbers below and, importantly, verify that the generated Elasticsearch documents are byte-for-byte identical (SHA-256 over every document) with and without the patch. Setup, once, in a KTD with Elasticsearch (stock sample data works; 10,000 generated records give steadier numbers - gen_records.pl + bulkmarcimport.pl, see README): mkdir -p _bench && cp <extracted files> _bench/ Test plan for this bug 1. prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t (includes the round-trip / over-length fallback subtests) 2. Before/after, with this patch applied on a branch: _bench/ab_bench.sh main HEAD Expect: "output identical to baseline" = yes at every batch size. This is the important check for this patch. 3. Fallback still works: index a record with a single field longer than 9999 bytes (or a record over 99999 bytes in total); it must be stored as MARCXML (ElasticsearchMARCFormat fallback), while a normal record is stored as base64ISO2709. Both must be retrievable from search results. 4. Optional, full reindex wall-clock: bash _bench/bench_rebuild.sh before / after. Our results (10,436 generated records, KTD + ES 9, one core, best of 3 interleaved rounds, spread 0-4%), decode + mapping, ms per record: batch 1: 24.8 -> 24.5 (1.01x) batch 10: 6.6 -> 6.3 (1.05x) batch 500: 4.6 -> 4.3 (1.08x) full rebuild_elasticsearch.pl: 103.8 s -> 100.4 s (1.03x) Honest note: our earlier profile suggested the round trip was 20-25% of mapping CPU, but end to end we only measure a few percent on this data set. That may be down to our synthetic records (small, with few fields); records with many large fields may benefit more. Please check this one specifically, and please also note the dependency discussion: this patch's rationale mentions bug 35104 (separator stripping on store), of which only the Metadata::Error classes are currently on main; the patch applies and is safe without it because of the separator check it adds. -- You are receiving this mail because: You are watching all bug changes.