https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39694 Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #181227|0 |1 is obsolete| | --- Comment #3 from Martin Renvoize (ashimema) <martin.renvoize@openfifth.co.uk> --- Created attachment 182176 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=182176&action=edit Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size This patch makes the indexer poll the DB in configurable batches. Pretty much like Rabbit does. The only change is that it will now pass a LIMIT <batch_size> to the query. To test: 0. Have KTD started with ES (skip if you have an instance running): $ ktd --es7 up -d 1. Stop the indexer and rabbit: $ ktd --shell k$ sudo koha-es-indexer --stop kohadev k$ sudo service rabbitmq-server stop 2. On a separate terminal, watch the queue: k$ watch -n 0.5 'echo "SELECT COUNT(*) FROM background_jobs WHERE status=\"new\"" | koha-mysql kohadev' => SUCCESS: It should be empty 3. Add a bunch of things to index (this works with default KTD data): k$ for i in $(seq 1 438; seq 1 438) ; do echo "INSERT INTO background_jobs (status,queue,data) VALUES ('new','elastic_index','{\"record_server\":\"biblioserver\",\"record_ids\":[$i]}')" | koha-mysql kohadev ; done 4. Start the ES indexer: k$ sudo koha-es-indexer --start kohadev 5. Look at the queue size => FAIL: It shrinks too fast [1] 6. Apply this patch 7. Restart the indexer k$ perl misc/workers/es_indexer_daemon.pl & 8. Repeat 3 9. Look at the queue size: => SUCCESS: It shrinks 'slowly' 10. Sign off :-D [1] This is a problem in certain scenarios, not in all of them. This is the best I come up with in order to test the behavior change. Anyone can set a new size of their taste. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> -- You are receiving this mail because: You are watching all bug changes.