Have you ever wondered why some programs simply stop when the server is very busy? There could be an error in the code causing the failure, or it could be a victim of kernel assassination!
The Linux kernel has some strange policies regarding how to manage RAM resources – it will allow the system to allocate more memory than is available – making the optimistic assumption that even when an application has requested a specific amount of memory, it won’t actually use it all.
Unfortunately, there are times when every machine gets very busy, and RAM can get used up incredibly quickly. This is the moment when the kernel needs to start reclaiming some of the memory it’s promised to processes and applications. To do this it starts a process called OOM Killer (Out-Of-Memory Killer) – guess what it does?
The Kernel anticipates this situation, keeping a running score of the system, by which it decides what to kill first. Will it be an scientific app that has been running for week and could loose some important calculations? This sounds important…. Then maybe system logger or httpd? Or maybe even better: sshd! It doesn’t seem to do a lot!…
If the kernel is left to make it’s own triage decisions, you might find yourself in trouble as it kills off processes to try and reclaim memory – sometimes making the situation worse.
An amusing description of this concept can be read here: http://lwn.net/Articles/104185/.
Thankfully, we are not entirely left at the mercy of the kernel, and there is way to adjust how the RAM will be managed (since kernel 2.6)
Here you can change the way ram is handled:
/proc/sys/vm/overcommit_memory
and it can be set to one of three values:
0: heuristic overcommit (this is the default)
1: always overcommit, never check
2: always check, never overcommit
And if you decide to go the safest option, set it to 2 and then adjust the ratio option too:
/proc/sys/vm/overcommit_ratio
The value in this file will determine the total virtual address space in the system (default is 50) and it works like this:
total virtual address space = (SS + RAM*(r/100))
Note – where SS is the size of the swap space, and RAM is the size of the physical memory, and r is the contents of the file /proc/sys/vm/overcommit_ratio.
If this is set to 100, the system should be able to use all of the RAM but not overcommit any.
As a final word, you may be wondering what processes will get killed first, and how to find it out? This little command should tell you what are your system’s favourites to kill as first – higher score = more likely to be terminated. Run this:
echo -e "SCORE\tPID\tNAME";for pid in $(ps ax|awk '{print $1}');do s=$(cat /proc/$pid/oom_score 2>/dev/null); p=$(ps ax|grep $pid|head -1|awk '{print $5}');echo -e "$s\t$pid\t$p"; done|sort -nr
There is also a file that will let you influence this decision and is located here:
/proc/[pid]/oom_adj
Valid values are in the range -16 to +15, plus the special value -17, which disables OOM-killing altogether for this process. A positive score increases the likelihood of this process being killed by the OOM-killer; a negative score decreases the likelihood.
The original score from the kernel is kept here:
/proc/[pid]/oom_score
More information on OOM Killer and score can be found here: http://www.kernel.org/doc/man-pages/online/pages/man5/proc.5.html.
