https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42879 --- Comment #2 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 208523 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=208523&action=edit Benchmark scripts for full-reindex / batch indexing (bug 42881 family) Clarifying the goal of this work (bug 42881 family): the target is infrastructure-facing - the time and CPU a FULL REINDEX (misc/search_tools/rebuild_elasticsearch.pl) and LARGE BATCH re-indexes (batch record modification, imports; these enqueue index jobs of up to chunk_size = 5000 records) consume. It is not about how quickly a cataloguer sees one edit: in KTD we measured the edit-to-searchable time through the stock es_indexer_daemon and it is dominated by job dispatch (roughly one job per second), so these patches do not change it. I am attaching a small benchmark package (bug42881-es-indexing-bench.tar.gz, see README.txt inside) so testers can reproduce the before/after numbers below and, importantly, verify that the generated Elasticsearch documents are byte-for-byte identical (SHA-256 over every document) with and without the patch. Setup, once, in a KTD with Elasticsearch (stock sample data works; 10,000 generated records give steadier numbers - gen_records.pl + bulkmarcimport.pl, see README): mkdir -p _bench && cp <extracted files> _bench/ Test plan for this bug 1. prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t 2. Before/after, with this patch applied on a branch: _bench/ab_bench.sh main HEAD Expect: "output identical to baseline" = yes; the plugin_methods lookup happens once per batch instead of once per record (DB statements per record fall by 1 at batch 500 and 10; at batch 1 there is still one lookup per call). 3. Plugins still work: install a plugin implementing the elasticsearch_to_document hook (adding a field to the document), reindex a record, and confirm the field is present in the ES document. Then disable the plugin (or plugins entirely) and confirm indexing still works and the field is gone. 4. Optional, full reindex wall-clock: bash _bench/bench_rebuild.sh before / after. Our results (10,436 generated records, KTD + ES 9, one core, best of 3 interleaved rounds, spread 0-4%), decode + mapping, ms per record: batch 1: 24.8 -> 23.6 (1.05x) batch 10: 6.6 -> 5.0 (1.34x) batch 500: 4.6 -> 2.9 (1.59x) full rebuild_elasticsearch.pl: 103.8 s -> 86.1 s (1.21x) As with 42878 this removes a per-record database round trip, so we expect a larger effect against a networked database. -- You are receiving this mail because: You are watching all bug changes.