[Koha-bugs] [Bug 22417] Add a task queue

[bugzilla-daemon at bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22417

--- Comment #322 from David Cook <dcook at prosentient.com.au> ---
(In reply to Jonathan Druart from comment #320)
> Same report on IRC with a possible solution:
> < kohaputti> seems to be around 4-10% on my machine more or less constantly
> < kohaputti> yes, it is beam.smp
> < kohaputti> maybe there is some configration we could change
> < kohaputti> (4-10% on one core)
> < kohaputti> just installed the rabbitmq stuff because I noticed reset_all
> needed it
> < kohaputti> I have the latest version of Koha master as of ~1 hour
> < kohaputti> lots of sched_yield() syscalls
> < kohaputti> when I look with strace
> < kohaputti> https://www.rabbitmq.com/runtime.html#cpu-reduce-idle-usage
> < kohaputti> adding RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+sbwt none" to the
> env vars seemed to do the trick
> < kohaputti> now the load is around 1-2% constantly
> < Joubu> which version of rabbitmq? which os?
> < kohaputti> seems to be debian 9 (from koha-testing-docker)
> < kohaputti> version of rabbitmq-server is 3.6.6-1
> 

I'm using Ubuntu 18.04 and RabbitMQ 3.6.10-1ubuntu0.1 in production. I think
Erlang is version 1:20.2.2+dfsg-1ubuntu2 (as per erlang-base)... and erts 9.2.
(I think this might be the relevant documentation:
http://erlang.org/documentation/doc-9.2/doc/). 

Environment:
USER=rabbitmqHOME=/var/lib/rabbitmqOLDPWD=/ERL_LIBS=/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/pluginsEMU=beamPROGNAME=erlRABBITMQ_CONFIG_FILE=/etc/rabbitmq/rabbitmqBINDIR=/usr/lib/erlang/erts-9.2/binLOGNAME=rabbitmqJOURNAL_STREAM=9:2636PATH=/usr/lib/erlang/erts-9.2/bin:/usr/lib/erlang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/binINVOCATION_ID=e123641da5f6417888a333ac9927f11cLANG=en_AU.UTF-8ROOTDIR=/usr/lib/erlangERL_MAX_ETS_TABLES=50000PWD=/var/lib/rabbitmq

Process:
/usr/lib/erlang/erts-9.2/bin/beam.smp -W w -A 128 -P 1048576 -t 5000000 -stbt
db -zdbbl 32000 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home
/var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/ebin
-noshell -noinput -s rabbit boot -sname rabbit at server -boot start_sasl -kernel
inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl
sasl_error_logger false -rabbit error_logger
{file,"/var/log/rabbitmq/rabbit at server.log"} -rabbit sasl_error_logger
{file,"/var/log/rabbitmq/rabbit at server-sasl.log"} -rabbit enabled_plugins_file
"/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir
"/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/plugins"
-rabbit plugins_expand_dir
"/var/lib/rabbitmq/mnesia/rabbit at server-plugins-expand" -os_mon start_cpu_sup
false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir
"/var/lib/rabbitmq/mnesia/rabbit at server" -kernel inet_dist_listen_min 25672
-kernel inet_dist_listen_max 25672

Over 2.5 days, it's used about 1hr 10 minutes of CPU time for processing 0
messages.

CPU usage varies but it's more so in the 1-2% range. 

--

But... using strace I do see a tonne of sched_yield() syscalls. 

Kind of interesting reading Linus Torvald's 2020 indictment of sched_yield
(https://www.realworldtech.com/forum/?threadid=189711&curpostid=189752), and a
rude 2016 defense of sched_yield by someone
http://erlang.org/pipermail/erlang-questions/2016-March/088232.html. 

I've tried adding the following to /etc/rabbitmq/rabbitmq-env.conf as per
https://www.rabbitmq.com/runtime.html#cpu-reduce-idle-usage but it caused
RabbitMQ to fail to start up:

SERVER_ADDITIONAL_ERL_ARGS="+sbwt none +sbwtdcpu none +sbwtdio none"

However, the server would start with the following:

SERVER_ADDITIONAL_ERL_ARGS="+sbwt none"

Now I'm not seeing any sched_yield syscalls via strace. I do still see a lot of
clock_gettime, timerfd_settime, epoll_wait (with no timeout) and futex. Not
sure why it's checking the time so much. It looks to me like Erlang isn't
actually sleeping the schedulers, despite it saying it puts schedulers to sleep
if +sbwt is none. I'm guessing the clock_gettime is used to monitor each thread
to make sure it's not trying to do too much, which is a far cry from
sleeping... 

But overall, I'd say CPU usage is about 0-1% now. 

It may be worth noting that RabbitMQ says "The runtime assumes that it does not
share CPU resources with other tools or tenants. When that's the case, the
scheduling mechanism used can become very inefficient and result in significant
(up to several orders of magnitude) latency increase for certain operations.
This means that in most cases colocating RabbitMQ nodes with other tools or
applying CPU time slicing is highly discouraged and will result in suboptimal
performance." at https://www.rabbitmq.com/runtime.html#cpu-contention

Overall, I'm still not concerned.

-- 
You are receiving this mail because:
You are watching all bug changes.