Exited (137) means the container's main process was ended with SIGKILL. In this
test three different things produced it: the memory limit (out of memory), docker
kill, and docker stop on a process that ignored the stop signal.
docker inspect tells them apart: only the memory case shows
OOMKilled=true.
If OOMKilled is true and the container has a -m limit, raise it, or
make the program use less memory. If it is false, look for who stopped the container, or for a process that does
not exit on SIGTERM.
The error
PS> docker run --name fix9-run-oom -m 32m --memory-swap 32m alpine sh -c "x=a; i=0; while [ `$i -lt 24 ]; do x=`$x`$x; i=`$((i+1)); done; echo `${#x}"
(exit code 137)
...
PS> docker ps -a --filter name=fix9-run-oom --format "{{.Names}} {{.Status}}"
fix9-run-oom Exited (137) Less than a second ago
The shell doubles a string 24 times, to 16,777,216 characters, inside a 32 MB limit. The
container printed nothing and stopped with 137. The backticks are PowerShell's escape so that
sh receives $i and $x.
Why it happens
137 is 128 + 9: the process was ended by signal 9, SIGKILL, which it cannot catch. The exit code alone does not say who sent it. The three cases, each run on the same engine:
PS> docker inspect fix9-run-oom --format "ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}} Error={{.State.Error}}"
ExitCode=137 OOMKilled=true Error=
PS> docker kill fix9-run-kill
fix9-run-kill
PS> docker inspect fix9-run-kill --format "ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}"
ExitCode=137 OOMKilled=false
PS> docker run -d --name fix9-run-stop alpine sh -c "trap '' TERM; sleep 300 & wait"
...
PS> Measure-Command { docker stop -t 3 fix9-run-stop } | Select-Object -ExpandProperty TotalSeconds
3.4039356
PS> docker inspect fix9-run-stop --format "ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}"
ExitCode=137 OOMKilled=false
The third container ignores SIGTERM (trap '' TERM), so docker stop -t
3 waited the three seconds and then killed it: the stop took 3.4 seconds and the exit
code is 137. A stop that takes as long as its timeout and ends in 137 points at
a process that does not handle SIGTERM. The engine's event log also records the memory kill:
PS> docker events --since 5m --until 0s --filter container=fix9-run-oom --format "{{.Action}}"
create
attach
start
oom
die
The fix
For the out-of-memory case, the same command with a 256 MB limit finished and printed the string length:
PS> docker run --name fix9-run-ok -m 256m --memory-swap 256m alpine sh -c "x=a; i=0; while [ `$i -lt 24 ]; do x=`$x`$x; i=`$((i+1)); done; echo `${#x}"
16777216
PS> docker inspect fix9-run-ok --format "ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}"
ExitCode=0 OOMKilled=false
Raising the limit is the quick fix; the lasting one is finding what grows. For the
docker kill case there is nothing to fix in the container: find the script or
person that sent it. For the docker stop case, the program needs to exit when it
receives SIGTERM; the test container ignored it on purpose with trap '' TERM.
How it was reproduced
On Windows 11 with Docker Desktop, engine and client 29.9.3, from Windows PowerShell 5.1, with
the cached alpine image. One container hit a 32 MB limit
(-m 32m --memory-swap 32m), one was ended with docker kill, one
ignored SIGTERM and was stopped with docker stop -t 3. All three exited with 137;
only the first had OOMKilled=true and an oom event. The containers
are the test's own and were removed afterwards.
Frequently asked
- What does exit code 137 mean in Docker?
- The main process was ended by SIGKILL (128 + 9). In this test that came from the memory limit, from docker kill, and from docker stop after its timeout. docker inspect shows OOMKilled=true only for the memory case.
- How do I check if a Docker container was OOM killed?
- Run docker inspect with --format and the field .State.OOMKilled. It printed true for the container that hit its 32 MB limit and false for the killed and stopped ones.
- Why does docker stop end with exit code 137?
- The process did not exit on SIGTERM, so Docker killed it when the stop timeout ran out. In this test docker stop -t 3 took 3.4 seconds and gave 137.
More decoded errors in the Fixes category. A memory kill of a model server in a container: llama-server signal: killed.