Memcached was the process requesting more memory, not necessary the process using a lot of memory.
Looking through the logs you attached it seems Beam.smp (which I believe is RabbitMQ), MariaDB, and Koha Plack are the top three memory processes.  rebuild_zebra is also using a lot of paged memory, but its total vm usage isn't high; I'm not sure how that factors in here.  There may be processes (like Kernel processes) that aren't in that list, I'm not going to do the math to see if all the processes are listed.  I'm not even sure if the values here are bytes or memory pages.
The OOM Killer decided that Plack was the most expendable process, that's why it killed it.
Basically, all you can do is reduce memory usage or add memory.  I don't know if adding more swap space is a good solution, swap memory is a lot slower than physical ram, but it's also a lot cheaper.  Looks like all the swap space is allocated too, so it is being used as far as I can tell.
I might see if all of the Plack processes are in use, and reduce that if it's higher than the peak active requests (Maybe Apache can provide that number).  That's the plack_workers tag in the config file, and defaults to 2, so that only works if you have increased the number significantly past the default.  You can also look at the plack_max_requests tag, to recycle the workers quicker.  That may free up ram quicker, but spawning workers takes time.  That setting defaults to 50 (requests).



On Mon, Mar 10, 2025 at 10:27 AM Michael Kuhn via Koha-devel <koha-devel@lists.koha-community.org> wrote:
Hi

Our host "bib" owns 4 virtual CPU, 8 GB RAM and 18 GB swap space. It
runs on Debian GNU/Linux 12 with Koha 24.05.01 and around 78'000
bibliographic records.

Since some time Koha sporadically just stops (today already twice), in
such cases log file "/var/log/syslog" contains messages as attached at
the end of this e-mail (the "oom-killer" is not always invoked by
memcached). The messages say the host is out of memory so the Koha
process was killed... I have already increased the swap space from 2 GB
to 18 GB but to no avail.

How can we find out what is consuming so much memory causing Koha to get
killed? How can we avoid this?

Best wishes: Michael
--
Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis
Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz
T 0041 (0)61 261 55 61 · E mik@adminkuhn.ch · W www.adminkuhn.ch




2025-03-10T12:48:00.668764+01:00 bib kernel: [769596.762201] memcached
invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP),
order=0, oom_score_adj=0
2025-03-10T12:48:00.670036+01:00 bib kernel: [769596.762885] CPU: 3 PID:
2332685 Comm: memcached Not tainted 6.1.0-31-amd64 #1  Debian 6.1.128-1
2025-03-10T12:48:00.670038+01:00 bib kernel: [769596.763487] Hardware
name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014
2025-03-10T12:48:00.670039+01:00 bib kernel: [769596.764019] Call Trace:
2025-03-10T12:48:00.670039+01:00 bib kernel: [769596.764528]  <TASK>
2025-03-10T12:48:00.670040+01:00 bib kernel: [769596.765027]
dump_stack_lvl+0x44/0x5c
2025-03-10T12:48:00.670040+01:00 bib kernel: [769596.765539]
dump_header+0x4a/0x211
2025-03-10T12:48:00.670041+01:00 bib kernel: [769596.766045]
oom_kill_process.cold+0xb/0x10
2025-03-10T12:48:00.670041+01:00 bib kernel: [769596.766535]
out_of_memory+0x1fd/0x4c0
2025-03-10T12:48:00.670042+01:00 bib kernel: [769596.767046]
__alloc_pages_slowpath.constprop.0+0xc83/0xe40
2025-03-10T12:48:00.670042+01:00 bib kernel: [769596.767537]
__alloc_pages+0x305/0x330
2025-03-10T12:48:00.670043+01:00 bib kernel: [769596.768014]
folio_alloc+0x17/0x50
2025-03-10T12:48:00.670043+01:00 bib kernel: [769596.768499]
__filemap_get_folio+0x155/0x340
2025-03-10T12:48:00.670044+01:00 bib kernel: [769596.768967]
filemap_fault+0x139/0x910
2025-03-10T12:48:00.670044+01:00 bib kernel: [769596.769395]  ?
filemap_map_pages+0x153/0x700
2025-03-10T12:48:00.670044+01:00 bib kernel: [769596.769820]
__do_fault+0x33/0x110
2025-03-10T12:48:00.670045+01:00 bib kernel: [769596.770303]
do_fault+0x1b9/0x410
2025-03-10T12:48:00.670157+01:00 bib kernel: [769596.770734]
__handle_mm_fault+0x660/0xfa0
2025-03-10T12:48:00.670298+01:00 bib kernel: [769596.771189]
handle_mm_fault+0xdb/0x2d0
2025-03-10T12:48:00.670299+01:00 bib kernel: [769596.771594]
do_user_addr_fault+0x191/0x550
2025-03-10T12:48:00.670300+01:00 bib kernel: [769596.771982]
exc_page_fault+0x70/0x170
2025-03-10T12:48:00.670300+01:00 bib kernel: [769596.772411]
asm_exc_page_fault+0x22/0x30
2025-03-10T12:48:00.670301+01:00 bib kernel: [769596.772829] RIP:
0033:0x557b12827140
2025-03-10T12:48:00.670302+01:00 bib kernel: [769596.773207] Code:
Unable to access opcode bytes at 0x557b12827116.
2025-03-10T12:48:00.670302+01:00 bib kernel: [769596.773589] RSP:
002b:00007ffc7f74a9c8 EFLAGS: 00010246
2025-03-10T12:48:00.670303+01:00 bib kernel: [769596.773991] RAX:
00007fc1802bd440 RBX: 0000557b44749ac0 RCX: 0000000000000669
2025-03-10T12:48:00.670303+01:00 bib kernel: [769596.774387] RDX:
0000000000000000 RSI: 0000000000000001 RDI: 00000000ffffffff
2025-03-10T12:48:00.670303+01:00 bib kernel: [769596.774723] RBP:
0000000000000001 R08: 0000557b12827140 R09: 00007fc1802dd000
2025-03-10T12:48:00.670304+01:00 bib kernel: [769596.775111] R10:
0000000000000000 R11: 0000000000000000 R12: 0000000000000001
2025-03-10T12:48:00.670305+01:00 bib kernel: [769596.775506] R13:
0000000000000000 R14: 000000007fffffff R15: 0000557b1285cb80
2025-03-10T12:48:00.670305+01:00 bib kernel: [769596.775831]  </TASK>
2025-03-10T12:48:00.670305+01:00 bib kernel: [769596.776226] Mem-Info:
2025-03-10T12:48:00.670365+01:00 bib kernel: [769596.776592]
active_anon:550829 inactive_anon:1391840 isolated_anon:0
2025-03-10T12:48:00.670366+01:00 bib kernel: [769596.776592]
active_file:0 inactive_file:94 isolated_file:0
2025-03-10T12:48:00.670366+01:00 bib kernel: [769596.776592]
unevictable:0 dirty:0 writeback:0
2025-03-10T12:48:00.670367+01:00 bib kernel: [769596.776592]
slab_reclaimable:9935 slab_unreclaimable:13188
2025-03-10T12:48:00.670367+01:00 bib kernel: [769596.776592]
mapped:1134 shmem:5697 pagetables:5740
2025-03-10T12:48:00.670367+01:00 bib kernel: [769596.776592]
sec_pagetables:0 bounce:0
2025-03-10T12:48:00.670368+01:00 bib kernel: [769596.776592]
kernel_misc_reclaimable:0
2025-03-10T12:48:00.670368+01:00 bib kernel: [769596.776592]  free:25298
free_pcp:0 free_cma:0
2025-03-10T12:48:00.670368+01:00 bib kernel: [769596.789186] Node 0
active_anon:2203316kB inactive_anon:5567360kB active_file:80kB
inactive_file:156kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB mapped:4536kB dirty:0kB writeback:0kB shmem:22788kB
shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 876544kB writeback_tmp:0kB
kernel_stack:9056kB pagetables:22960kB sec_pagetables:0kB
all_unreclaimable? no
2025-03-10T12:48:00.670369+01:00 bib kernel: [769596.790069] Node 0 DMA
free:14336kB boost:0kB min:128kB low:160kB high:192kB
reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB
active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB
present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB
local_pcp:0kB free_cma:0kB
2025-03-10T12:48:00.670370+01:00 bib kernel: [769596.790924]
lowmem_reserve[]: 0 2899 7843 7843 7843
2025-03-10T12:48:00.670370+01:00 bib kernel: [769596.791252] Node 0
DMA32 free:44376kB boost:0kB min:24936kB low:31168kB high:37400kB
reserved_highatomic:0KB active_anon:842432kB inactive_anon:2119316kB
active_file:0kB inactive_file:84kB unevictable:0kB writepending:0kB
present:3129196kB managed:3018408kB mlocked:0kB bounce:0kB free_pcp:0kB
local_pcp:0kB free_cma:0kB
2025-03-10T12:48:00.670370+01:00 bib kernel: [769596.792175]
lowmem_reserve[]: 0 0 4943 4943 4943
2025-03-10T12:48:00.670371+01:00 bib kernel: [769596.792455] Node 0
Normal free:42480kB boost:0kB min:42516kB low:53144kB high:63772kB
reserved_highatomic:0KB active_anon:1360884kB inactive_anon:3447664kB
active_file:120kB inactive_file:0kB unevictable:0kB writepending:0kB
present:5242880kB managed:5062604kB mlocked:0kB bounce:0kB free_pcp:0kB
local_pcp:0kB free_cma:0kB
2025-03-10T12:48:00.670371+01:00 bib kernel: [769596.793398]
lowmem_reserve[]: 0 0 0 0 0
2025-03-10T12:48:00.670372+01:00 bib kernel: [769596.793716] Node 0 DMA:
0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
1*2048kB (M) 3*4096kB (M) = 14336kB
2025-03-10T12:48:00.670372+01:00 bib kernel: [769596.794404] Node 0
DMA32: 115*4kB (UME) 68*8kB (UME) 110*16kB (UME) 94*32kB (UME) 78*64kB
(UME) 56*128kB (UE) 40*256kB (UME) 15*512kB (E) 3*1024kB (UE) 3*2048kB
(E) 0*4096kB = 45068kB
2025-03-10T12:48:00.670373+01:00 bib kernel: [769596.795118] Node 0
Normal: 412*4kB (UME) 307*8kB (UME) 166*16kB (UME) 167*32kB (UME)
83*64kB (UME) 50*128kB (UME) 23*256kB (E) 13*512kB (ME) 7*1024kB (ME)
0*2048kB 0*4096kB = 43528kB
2025-03-10T12:48:00.670373+01:00 bib kernel: [769596.795916] Node 0
hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
2025-03-10T12:48:00.670373+01:00 bib kernel: [769596.796271] Node 0
hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
2025-03-10T12:48:00.670374+01:00 bib kernel: [769596.796713] 155111
total pagecache pages
2025-03-10T12:48:00.670374+01:00 bib kernel: [769596.797122] 149255
pages in swap cache
2025-03-10T12:48:00.670374+01:00 bib kernel: [769596.797463] Free swap
= 0kB
2025-03-10T12:48:00.670375+01:00 bib kernel: [769596.797848] Total swap
= 1998844kB
2025-03-10T12:48:00.670922+01:00 bib kernel: [769596.798238] 2097017
pages RAM
2025-03-10T12:48:00.670926+01:00 bib kernel: [769596.798577] 0 pages
HighMem/MovableOnly
2025-03-10T12:48:00.670926+01:00 bib kernel: [769596.798954] 72924 pages
reserved
2025-03-10T12:48:00.671386+01:00 bib kernel: [769596.799356] 0 pages
hwpoisoned
2025-03-10T12:48:00.671388+01:00 bib kernel: [769596.799693] Tasks state
(memory values in pages):
2025-03-10T12:48:00.671388+01:00 bib kernel: [769596.800070] [  pid  ]
uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
2025-03-10T12:48:00.671389+01:00 bib kernel: [769596.800463] [    274]
   0   274    30075       22   241664      274          -250 systemd-journal
2025-03-10T12:48:00.671389+01:00 bib kernel: [769596.800839] [    295]
   0   295     6770      278    86016      221         -1000 systemd-udevd
2025-03-10T12:48:00.671390+01:00 bib kernel: [769596.801224] [    524]
997   524    22521      154    77824       82             0 systemd-timesyn
2025-03-10T12:48:00.671390+01:00 bib kernel: [769596.801603] [    609]
   0   609     2372       35    53248       28             0 cron
2025-03-10T12:48:00.671390+01:00 bib kernel: [769596.801997] [    610]
100   610     2011      143    61440        8          -900 dbus-daemon
2025-03-10T12:48:00.671391+01:00 bib kernel: [769596.802387] [    612]
   0   612    55447      184    81920      116             0 rsyslogd
2025-03-10T12:48:00.671391+01:00 bib kernel: [769596.802749] [    613]
   0   613     4185      203    73728       73             0 systemd-logind
2025-03-10T12:48:00.671391+01:00 bib kernel: [769596.803147] [    629]
   0   629      895       40    49152        3             0 atd
2025-03-10T12:48:00.671392+01:00 bib kernel: [769596.803558] [    666]
   0   666     1499      202    53248       45             0 dhclient
2025-03-10T12:48:00.671392+01:00 bib kernel: [769596.803913] [    715]
   0   715     1435      100    49152      105             0 dhclient
2025-03-10T12:48:00.671393+01:00 bib kernel: [769596.804313] [    747]
104   747     2208      550    61440       34             0 epmd
2025-03-10T12:48:00.671393+01:00 bib kernel: [769596.804716] [    753]
   0   753   173229     3280   430080      880             0 fail2ban-server
2025-03-10T12:48:00.671393+01:00 bib kernel: [769596.805078] [    774]
106   774   920414    28673  1294336    23431          -500 beam.smp
2025-03-10T12:48:00.671393+01:00 bib kernel: [769596.805494] [    797]
   0   797     2188       20    53248        2             0 agetty
2025-03-10T12:48:00.671394+01:00 bib kernel: [769596.806006] [    812]
   0   812     3859      213    65536      120         -1000 sshd
2025-03-10T12:48:00.671394+01:00 bib kernel: [769596.806567] [    841]
106   841      616       19    53248        6          -500 erl_child_setup
2025-03-10T12:48:00.671395+01:00 bib kernel: [769596.807045] [   1774]
   0  1774    10665      108    69632       53             0 master
2025-03-10T12:48:00.671395+01:00 bib kernel: [769596.807442] [   1779]
102  1779    10700      127    81920       48             0 qmgr
2025-03-10T12:48:00.671396+01:00 bib kernel: [769596.807848] [   2023]
106  2023      848       11    45056       14          -500 inet_gethost
2025-03-10T12:48:00.671396+01:00 bib kernel: [769596.808192] [   2026]
106  2026      848       24    45056        8          -500 inet_gethost
2025-03-10T12:48:00.671397+01:00 bib kernel: [769596.808560] [   2034]
106  2034      644       14    40960       14          -500 sh
2025-03-10T12:48:00.671397+01:00 bib kernel: [769596.808953] [   3262]
103  3262  2058793   518769  7716864   392357             0 mariadbd
2025-03-10T12:48:00.671397+01:00 bib kernel: [769596.809287] [ 392666]
102 392666    12109      128    90112      225             0 tlsmgr
2025-03-10T12:48:00.671398+01:00 bib kernel: [769596.809647] [2332677]
   0 2332677     3456      434    61440      303             0 apache2
2025-03-10T12:48:00.671398+01:00 bib kernel: [769596.810031] [2332685]
105 2332685   107470     2677   159744     2787             0 memcached
2025-03-10T12:48:00.671399+01:00 bib kernel: [769596.810358] [2332735]
1003 2332735     2128       21    49152       15             0 daemon
2025-03-10T12:48:00.671399+01:00 bib kernel: [769596.810703] [2332737]
1003 2332737    14826      161   110592      256             0 zebrasrv
2025-03-10T12:48:00.671399+01:00 bib kernel: [769596.811075] [2332802]
1003 2332802     6211     2612    94208      287             0 starman
master
2025-03-10T12:48:00.671400+01:00 bib kernel: [769596.811400] [2332846]
1003 2332846     2128       37    49152        0             0 daemon
2025-03-10T12:48:00.671400+01:00 bib kernel: [769596.811734] [2332847]
1003 2332847    29751    13047   221184      414             0
background_jobs
2025-03-10T12:48:00.671401+01:00 bib kernel: [769596.812076] [2332875]
1003 2332875     2128       21    45056       14             0 daemon
2025-03-10T12:48:00.671401+01:00 bib kernel: [769596.812396] [2332877]
1003 2332877    41320    19112   323584     4526             0
background_jobs
2025-03-10T12:48:00.671401+01:00 bib kernel: [769596.812718] [2332902]
1003 2332902     2128       13    49152       22             0 daemon
2025-03-10T12:48:00.671402+01:00 bib kernel: [769596.813044] [2332904]
1003 2332904    73759    36200   540672     6622             0
rebuild_zebra.p
2025-03-10T12:48:00.671402+01:00 bib kernel: [769596.813369] [1011996]
102 1011996    10671      154    73728        0             0 pickup
2025-03-10T12:48:00.671402+01:00 bib kernel: [769596.813651] [1014020]
  33 1014020     3517      422    61440      316             0 apache2
2025-03-10T12:48:00.671403+01:00 bib kernel: [769596.813976] [1015250]
  33 1015250     3517      426    61440      311             0 apache2
2025-03-10T12:48:00.671403+01:00 bib kernel: [769596.814307] [1018044]
  33 1018044     3517      472    61440      265             0 apache2
2025-03-10T12:48:00.671403+01:00 bib kernel: [769596.814583] [1018109]
  33 1018109     3517      465    61440      272             0 apache2
2025-03-10T12:48:00.671404+01:00 bib kernel: [769596.814882] [1018113]
  33 1018113     3517      473    61440      265             0 apache2
2025-03-10T12:48:00.671404+01:00 bib kernel: [769596.815171] [1020113]
  33 1020113     3517      473    61440      264             0 apache2
2025-03-10T12:48:00.671404+01:00 bib kernel: [769596.815506] [1024635]
  33 1024635     3517      474    61440      264             0 apache2
2025-03-10T12:48:00.671405+01:00 bib kernel: [769596.815796] [1026139]
  33 1026139     3517      474    61440      264             0 apache2
2025-03-10T12:48:00.671405+01:00 bib kernel: [769596.816089] [1026170]
  33 1026170     3517      473    61440      264             0 apache2
2025-03-10T12:48:00.671405+01:00 bib kernel: [769596.816378] [1026641]
1003 1026641  1185289  1122584  9445376    29621             0
/usr/share/koha
2025-03-10T12:48:00.671406+01:00 bib kernel: [769596.816688] [1026652]
  33 1026652     3517      473    61440      264             0 apache2
2025-03-10T12:48:00.671406+01:00 bib kernel: [769596.816972] [1027075]
1003 1027075    99496    34733   749568    33424             0
/usr/share/koha
2025-03-10T12:48:00.671407+01:00 bib kernel: [769596.817289] [1027426]
1003 1027426     3594      624    65536      241             0 apache2
2025-03-10T12:48:00.671416+01:00 bib kernel: [769596.817615] [1027453]
1003 1027453     3592      669    61440      240             0 apache2
2025-03-10T12:48:00.671416+01:00 bib kernel: [769596.817890] [1027749]
1003 1027749    16877      179   114688     2268             0 zebrasrv
2025-03-10T12:48:00.671416+01:00 bib kernel: [769596.818218] [1027761]
1003 1027761     3538      563    61440      244             0 apache2
2025-03-10T12:48:00.671417+01:00 bib kernel: [769596.818501] [1027769]
1003 1027769    16966     2354   114688      229             0 zebrasrv
2025-03-10T12:48:00.671417+01:00 bib kernel: [769596.818812] [1027770]
1003 1027770     3523      559    61440      241             0 apache2
2025-03-10T12:48:00.671417+01:00 bib kernel: [769596.819085]
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/koha-common.service,task=/usr/share/koha,pid=1026641,uid=1003
2025-03-10T12:48:00.671418+01:00 bib kernel: [769596.819717] Out of
memory: Killed process 1026641 (/usr/share/koha) total-vm:4741156kB,
anon-rss:4490332kB, file-rss:4kB, shmem-rss:0kB, UID:1003
pgtables:9224kB oom_score_adj:0
2025-03-10T12:48:00.721396+01:00 bib systemd[1]: koha-common.service: A
process of this unit has been killed by the OOM killer.
2025-03-10T12:48:03.702481+01:00 bib koha-common[1027819]: Stopping
Plack daemon for bib:.
2025-03-10T12:48:03.755927+01:00 bib koha-common[1027861]: Stopping Koha
worker daemon for bib (default):.
2025-03-10T12:48:03.789372+01:00 bib koha-common[1027886]: Stopping Koha
worker daemon for bib (long_tasks):.
2025-03-10T12:48:03.825612+01:00 bib koha-common[1027911]: Stopping Koha
indexing daemon for bib:.
2025-03-10T12:48:05.057975+01:00 bib systemd[1]: koha-common.service:
Failed with result 'oom-kill'.
2025-03-10T12:48:05.058217+01:00 bib systemd[1]: koha-common.service:
Consumed 2d 22h 25min 610ms CPU time.
_______________________________________________
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/


--
Michael Hafen
Washington County School District Technology Department
Systems & Security Analyst