https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=42881 --- Comment #1 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 208525 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=208525&action=edit Benchmark scripts for full-reindex / batch indexing (bug 42881 family) Clarifying the goal of this work (bug 42881 family): the target is infrastructure-facing - the time and CPU a FULL REINDEX (misc/search_tools/rebuild_elasticsearch.pl) and LARGE BATCH re-indexes (batch record modification, imports; these enqueue index jobs of up to chunk_size = 5000 records) consume. It is not about how quickly a cataloguer sees one edit: in KTD we measured the edit-to-searchable time through the stock es_indexer_daemon and it is dominated by job dispatch (roughly one job per second), so these patches do not change it. I am attaching a small benchmark package (bug42881-es-indexing-bench.tar.gz, see README.txt inside) so testers can reproduce the before/after numbers below and, importantly, verify that the generated Elasticsearch documents are byte-for-byte identical (SHA-256 over every document) with and without the patch. Setup, once, in a KTD with Elasticsearch (stock sample data works; 10,000 generated records give steadier numbers - gen_records.pl + bulkmarcimport.pl, see README): mkdir -p _bench && cp <extracted files> _bench/ Summary across the four dependencies, stacked (all applied; the output digest is identical to main in every run): Decode + mapping, ms per record, main -> all four: batch 1: 24.8 -> 3.6 (6.9x) batch 10: 6.6 -> 1.6 (4.1x) batch 500: 4.6 -> 1.4 (3.4x) DB statements per record: 3.0 / 2.1 / 2.0 -> 2.0 / 0.2 / 0.01 Real rebuild_elasticsearch.pl -d -r -b (1 process, 10,436 biblios, KTD + ES 9, best of 2, interleaved): main 103.8 s (100 biblios/s) -> all four 70.9 s (147 biblios/s) = 1.46x individually: 42877 1.01x, 42878 1.16x, 42879 1.21x, 42880 1.03x The remaining time is DB reads, MARCXML decoding and the Elasticsearch bulk requests. Per-bug test plans and numbers are in the comments on 42877-42880. Known caveat when testing the whole set: the four patches' unit tests textually conflict when stacked (each adds subtests at the same place in t/db_dependent/Koha/SearchEngine/Elasticsearch.t, and 42878/42879 touch adjacent lines of marc_records_to_documents), so whichever is pushed second will need a trivial rebase. -- You are receiving this mail because: You are watching all bug changes.