Log Dejargonizer

Linux kernel · Event Out of memory: Killed process

Out of memory: the kernel killed a process to stay alive

Linux ran out of memory and picked something to kill. It does not kill whatever caused the problem — it kills whatever scores highest, which is usually the largest process. That is why the database dies when a backup script leaks memory.

Also written as Out of memory: Killed processoom-killer invokedlinux oom killer logKilled process total-vm anon-rss

What it means for you

On a personal computer

Your computer ran out of memory and Linux shut a program down to keep going. Nothing is damaged. If it keeps happening, either something is using far more memory than it should or the machine genuinely needs more.

For an analyst

Read the whole block, not the summary line — the kernel prints a table of every candidate process with its memory usage and score just above the kill, and that table tells you what was actually consuming the memory, which is frequently not what got killed. Distinguish a system-wide OOM from a cgroup OOM: under systemd or a container runtime, 'Memory cgroup out of memory' means one unit hit its own limit while the machine had memory to spare, and the fix is the limit, not the machine. Also worth knowing that Linux overcommits by default, so allocation succeeds long before memory exists and the failure surfaces at first touch rather than at malloc.

What it looks like

Sanitised. Addresses come from the RFC 5737 documentation ranges.

Sample
kernel: node invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
kernel: Tasks state (memory values in pages):
kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
kernel: [   1187]   999  1187   412883   198442      1802240        0             0 postgres
kernel: [   4021]  1000  4021  1204553   882014      7442432        0             0 node
kernel: Out of memory: Killed process 1187 (postgres) total-vm:1651532kB, anon-rss:793768kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:1760kB oom_score_adj:0

The fields that decide it

Everything else in the log line is context.

Killed process name and PID
What was terminated. The victim, not necessarily the culprit.
total-vm
How much address space the process had reserved. Frequently enormous and frequently misleading — reserved is not used, and this number panics people unnecessarily.
anon-rss
Actual private memory resident in RAM. This is the number that matters when working out what was really consuming memory.
file-rss and shmem-rss
Memory backed by files and shared memory. Usually small, and reclaimable in a way anon-rss is not.
oom_score_adj
The tuning value applied to the process's score.
  • 0The default. No adjustment.
  • -1000Exempt — the kernel will not choose this process. Reserved for things whose death makes the situation worse.
  • A positive valueMade more likely to be killed. Often set deliberately on batch jobs so they die before services do.
The candidate table above the kill line
Not a single field, but the most useful part of the whole event. Every eligible process with its rss and score, printed immediately before the decision.
'Memory cgroup out of memory'
Whether this was a cgroup limit rather than the whole machine. Completely changes the diagnosis and the fix.

Ordinary reasons this happens

Most of the time it is one of these.

  • A machine sized for its normal load meeting an unusual but legitimate peak — a large import, a heavy report, a rebuild.
  • A container or systemd unit with a memory limit set lower than the job genuinely needs.
  • No swap configured, so the kernel has nowhere to push cold pages and reaches the killer sooner than expected.
  • A memory leak in ordinary application code, which is by far the most common underlying cause.
  • Many worker processes each individually reasonable, sized for a machine larger than this one.
  • A build or test job that briefly needs several gigabytes running alongside a service that needed the same memory.

When it is not ordinary

Cryptocurrency mining or another unauthorised workload consuming the machine's memory.

What gives it awayAn unfamiliar process high in the candidate table, often with a plausible-looking name, running from a temp or user-writable path.

ATT&CK T1496

Deliberate resource exhaustion to bring down a service.

What gives it awayRepeated OOM kills of the same service with the memory pressure coming from a process that appeared recently and has no owner.

ATT&CK T1499.001

A security agent killed as collateral, or as the intent.

What gives it awayAn agent or logging daemon appearing as the victim repeatedly, particularly if its oom_score_adj was recently changed from a protective value.

ATT&CK T1562.001

What to do next

  1. Scroll up. The summary line names the victim; the table above it names the consumers. Read the table.
  2. Sort that table by rss, not by total-vm. Reserved address space is not memory in use and it misleads constantly.
  3. Check whether the message says 'Memory cgroup out of memory'. If it does, one unit hit its own limit and the machine was fine — raise or remove the limit rather than adding RAM.
  4. Establish whether this is the first time. A single OOM under a known peak is capacity; a repeating one at increasing frequency is a leak.
  5. Check whether swap exists. A machine with no swap reaches the killer abruptly, with none of the slowdown that would otherwise warn you.
  6. Confirm what the victim was and whether it restarted. Under systemd, a killed service usually restarts, which hides the incident until the next one.
  7. If the top consumer is unfamiliar, find out what started it before assuming it is a leak — check its path, its parent, and when it first appeared.
  8. Set oom_score_adj to protect what genuinely must survive, but treat that as triage rather than a fix; the kernel will simply kill the next thing instead.

Queries to run

grep The kills themselves. Run the journal query below to get the tables that explain them. 
grep -i 'killed process' /var/log/syslog /var/log/messages 2>/dev/null | tail -20
grep The 40 lines before each kill are the candidate table, which is where the answer is. 
journalctl -k --since '7 days ago' | grep -B 40 'Out of memory' | less
grep Human-readable timestamps, and it separates cgroup kills from machine-wide ones at a glance. 
dmesg -T | grep -E 'oom-kill|Out of memory|Memory cgroup out of memory'
grep Run this when the message mentioned a cgroup — it shows whether the unit's own limit was the constraint. 
systemctl show <unit> -p MemoryMax -p MemoryHigh -p MemoryCurrent

Common questions

Why did the OOM killer kill my database instead of the process that leaked?

Because it chooses by score, and the score is dominated by how much memory a process is using right now. A database holding a large cache legitimately looks like the biggest consumer on the machine, so it gets chosen even when a smaller process caused the shortage by growing. The table printed above the kill shows every candidate and is the way to find the real cause.

What is the difference between total-vm and anon-rss?

total-vm is address space the process has reserved, which can be far larger than anything it actually uses — Linux hands out reservations freely. anon-rss is private memory genuinely resident in RAM. When working out what consumed the memory, anon-rss is the number to read; total-vm regularly causes people to blame the wrong process.

Will adding swap stop the OOM killer?

It delays it rather than preventing it, and the delay is genuinely useful because a machine under memory pressure with swap slows down noticeably first, which gives you a chance to notice. Without swap the kernel goes from fine to killing things with very little warning. Swap does not fix a leak, though — it only buys time.

What does 'Memory cgroup out of memory' mean?

One control group hit its own memory limit while the machine as a whole still had memory free. That is what container memory limits and systemd's MemoryMax produce. The fix is the limit or the workload inside it, not the size of the machine, and adding RAM to the host changes nothing.

How do I stop a specific process from being killed?

Write a negative value to /proc/<pid>/oom_score_adj, or set OOMScoreAdjust in its systemd unit, with -1000 making it exempt entirely. Use this sparingly: exempting a process does not create memory, it just means the kernel kills something else. If the exempted process is the one leaking, you have removed the only thing keeping the machine alive.

Read next

Mentioned by

Vendor documentation

Last reviewed 6 September 2026