Your patch window closed clean, apt says everything upgraded, and three weeks later a scanner still flags the box as exposed. The binary on disk was fixed. The process in RAM never got the memo.
I got bitten by exactly this gap on a runner of mine: systemd-logind was still executing a binary that dpkg had replaced more than three weeks earlier. proc(5) makes detection trivial if you know which file to poke, so I wrote a ~160 line Python script instead of babysitting needrestart prompts on twelve boxes. The whole thing walks /proc, no dependencies, stdlib only.
Why version checks lie to you
When a package manager “upgrades” a binary it does not touch the running process at all. It writes a new file and unlinks the old one. The kernel is happy to keep executing an unlinked inode as long as somebody still maps it. Every check you would normally run reads the disk, not memory:
| Check | What it actually reads | Says after upgrade, no restart |
|---|---|---|
apt list --upgradable | dpkg database (disk) | “nothing pending” |
nginx -v | the new file on disk | latest version |
/proc/PID/exe | the inode in memory | path, followed by (deleted) |
That (deleted) marker is the entire trick. If a process main binary (or any mapped library) resolves to an unlinked file, the running copy is stale, no matter how current the package database claims to be. The same signal is what lsof users grep for after deleting log files that a process still holds open.

The script
It scans every PID, flags deleted exe targets, and maps each hit to its systemd unit through /proc/PID/cgroup so the output tells you exactly what to restart instead of throwing PIDs at you. Two extra flags: --include-libs parses /proc/PID/maps for deleted shared objects, and --min-hours hides processes that started after your last reboot, which keeps the noise down on boxes where you restart nightly. The exit code is the point: 0 clean, 1 stale stuff found, which means a single crontab line is a complete monitoring solution.
#!/usr/bin/env python3
"""stale_procs.py -- report processes still running DELETED (upgraded) binaries.
The problem: you run apt upgrade, the package manager replaces /usr/sbin/foo,
but the running process is still executing the OLD binary from disk. The new
(one usually containing the security fixes) sits unused until someone
restarts the service. This script walks /proc, flags any process whose exe
link points at an unlinked file ("(deleted)"), also checks for deleted
shared libraries still mapped via /proc/PID/maps, maps every hit to its
systemd unit, and prints a sorted report.
Exit codes: 0 = all clean, 1 = stale processes found (cron-friendly),
2 = environment error (not Linux / no /proc).
"""
import os
import re
import sys
import argparse
SO_RE = re.compile(r"[^/]+\.so(\.[0-9][0-9.]*)?$")
from datetime import datetime
PROC_OK = os.path.isdir("/proc/1")
def proc_start_time(pid):
"""Boot-relative clock ticks -> wall-clock datetime of process start."""
try:
with open(f"/proc/{pid}/stat") as fh:
fields = fh.read().rsplit(") ", 1)[1].split()
ticks = int(fields[19]) # field 22: starttime
hz = os.sysconf("SC_CLK_TCK")
with open("/proc/uptime") as fh:
uptime = float(fh.read().split()[0])
now = datetime.now().timestamp()
return datetime.fromtimestamp(now - uptime + ticks / hz)
except (OSError, IndexError, ValueError):
return None
def owning_unit(pid):
"""Map pid -> systemd unit name using /proc/PID/cgroup (v1 + v2)."""
unit = "-"
try:
with open(f"/proc/{pid}/cgroup") as fh:
for line in fh:
_, _, path = line.strip().partition("::")
if not path: # cgroup v1: ctrl,_,path split on ':'
parts = line.strip().split(":")
path = parts[2] if len(parts) == 3 else ""
for chunk in reversed(path.split("/")):
if chunk.endswith((".service", ".scope", ".slice")):
if not chunk.startswith("user@"):
return chunk
unit = chunk
except OSError:
pass
return unit
def deleted_exe(pid):
"""Return (path, None) if the main binary is deleted, else (None, None)."""
try:
target = os.readlink(f"/proc/{pid}/exe")
except OSError:
return None, None
if target.endswith(" (deleted)"):
return target[: -len(" (deleted)")], None
return None, None
def deleted_libs(pid):
"""Return set of deleted shared-library paths still mapped by pid."""
libs = set()
try:
with open(f"/proc/{pid}/maps") as fh:
for line in fh:
if "(deleted)" not in line:
continue
parts = line.rstrip("\n").split(maxsplit=5)
if len(parts) == 6 and parts[5].endswith(" (deleted)"):
path = parts[5][: -len(" (deleted)")]
if SO_RE.search(path):
libs.add(path)
except OSError:
pass
return libs
def main():
ap = argparse.ArgumentParser(description="find processes running deleted binaries")
ap.add_argument("--include-libs", action="store_true",
help="also report deleted shared libraries still mapped")
ap.add_argument("--min-hours", type=float, default=0.0, metavar="H",
help="ignore processes younger than H hours")
args = ap.parse_args()
if not PROC_OK:
print("error: /proc not available, Linux only", file=sys.stderr)
return 2
rows = []
self_pid = os.getpid()
for entry in os.listdir("/proc"):
if not entry.isdigit() or int(entry) == self_pid:
continue
pid = int(entry)
binary, _ = deleted_exe(pid)
hits = []
if binary:
hits.append(("binary", binary))
if args.include_libs:
for lib in sorted(deleted_libs(pid)):
if lib != binary:
hits.append(("library", lib))
if not hits:
continue
started = proc_start_time(pid)
if args.min_hours and started:
age_h = (datetime.now() - started).total_seconds() / 3600
if age_h < args.min_hours:
continue
try:
with open(f"/proc/{pid}/comm") as fh:
comm = fh.read().strip()
except OSError:
comm = "?"
rows.append((owning_unit(pid), pid, comm, started, hits))
if not rows:
print("OK: no processes running deleted binaries")
return 0
print(f"{'UNIT':<28} {'PID':>7} {'TYPE':<8} {'BINARY/LIBRARY':<44} STARTED")
stale = 0
for unit, pid, comm, started, hits in sorted(rows, key=lambda r: r[0]):
for typ, path in hits:
ts = started.strftime("%Y-%m-%d %H:%M") if started else "?"
print(f"{unit:<28} {pid:>7} {typ:<8} {path:<44} {ts}")
stale += 1
print(f"\n{stale} stale mapping(s) across {len(rows)} process(es)."
" restart the units above to load the new binaries.")
return 1
if __name__ == "__main__":
sys.exit(main())Proving it catches something
Trust, but verify: I built two deliberate victims on a throwaway Ubuntu 26.04 test runner (all lab data, nothing production) and let the script hunt. The unit files live in /etc/systemd/system/:

# /etc/systemd/system/stale-agent.service
[Unit]
Description=lane demo agent (deleted after start)
[Service]
ExecStart=/tmp/lane-2026-11-17/demo/bin1/sleep 3600
# /etc/systemd/system/metricd.service
[Unit]
Description=lane demo metric collector (deleted lib after start)
[Service]
Environment=PYTHONPATH=/tmp/lane-2026-11-17/demo/libs
ExecStart=/usr/bin/python3 -c "import _sqlite3, time; time.sleep(3600)"
metricd imports a copy of _sqlite3.so that I made with sys.path pointing into the demo dir, so I can delete that exact file while the interpreter keeps mapping it. Start both, then pull the rug:
systemctl daemon-reload
systemctl start stale-agent metricd
cp /usr/bin/sleep /tmp/lane-2026-11-17/demo/bin1/sleep
cp /usr/lib/python3.14/lib-dynload/_sqlite3.cpython-314-x86_64-linux-gnu.so \
/tmp/lane-2026-11-17/demo/libs/
systemctl restart stale-agent metricd
rm -f /tmp/lane-2026-11-17/demo/bin1/sleep \
/tmp/lane-2026-11-17/demo/libs/_sqlite3.cpython-314-x86_64-linux-gnu.soFirst debugging story: cp /usr/bin/sleep $D/v1-agent felt cleaner for the demo, so I renamed the copy. The unit exploded instantly with coreutils: unknown program 'v1-agent'. Modern Ubuntu coreutils is a busybox-style multicall binary that dispatches on argv[0], so the copy has to keep the name sleep. Renamed it, and the trap worked.
Nov 17 04:20:00 runner-int v1-agent[2877048]: coreutils: unknown program 'v1-agent'
Nov 17 04:20:00 runner-int systemd[1]: stale-agent.service: Main process exited, code=exited, status=1/FAILUREThe default run caught the planted binary plus two entries I did not plant. Those PIDs and dates are from a live host, still running binaries that the Oct 25 package refresh replaced:
UNIT PID TYPE BINARY/LIBRARY STARTED
init.scope 1529 binary /usr/lib/systemd/systemd-executor 2026-10-25 07:46
stale-agent.service 2877150 binary /tmp/lane-2026-11-17/demo/bin1/sleep 2026-11-17 04:20
systemd-logind.service 1066 binary /usr/lib/systemd/systemd-logind 2026-10-25 07:45
3 stale mapping(s) across 3 process(es). restart the units above to load the new binaries.The --include-libs flag is where the second, better bug surfaced. First attempt, 215 lines of output, and metricd.service was not among them. My maps parser did line.split()[-1], which returns the bare token (deleted), not the path, because the pathname column and the deletion marker are whitespace-joined. Nothing ever ended with the string I was matching on, so every library hit silently vanished:
$ python3 stale_procs.py --include-libs | grep -c metricd
0
# fix: keep the pathname column whole, maxsplit=5, then test it
parts = line.rstrip("\n").split(maxsplit=5)
if len(parts) == 6 and parts[5].endswith(" (deleted)"):
path = parts[5][: -len(" (deleted)")]
if SO_RE.search(path):
libs.add(path)One more trap, visible in that code: filtering maps lines by “(deleted)” alone flagged /dev/zero, locale-archive, gconv caches and journald segment files. Those are mapped-but-deleted, sure, but they are not stale security patches. The SO_RE regex (“something that looks like libfoo.so or libfoo.so.6.1”) cut a noisy 295-row dump down to the 215 rows worth reading. After the fix, the planted library shows up clean:
$ grep -m2 "(deleted)" /proc/2877047/maps | head -2
7c1eb7a70000-7c1eb7a77000 r--p 00000000 00:24 942037 /tmp/lane-2026-11-17/demo/libs/_sqlite3.cpython-314-x86_64-linux-gnu.so (deleted)
7c1eb7a77000-7c1eb7a89000 r-xp 00007000 00:24 942037 /tmp/lane-2026-11-17/demo/libs/_sqlite3.cpython-314-x86_64-linux-gnu.so (deleted)
$ python3 stale_procs.py --include-libs | grep metricd
metricd.service 2877047 library /tmp/lane-2026-11-17/demo/libs/_sqlite3.cpython-314-x86_64-linux-gnu.so 2026-11-17 04:20Making it a cron check
For the quiet path the script prints one line and exits 0, so your alerting logic is just cron exit-code plumbing. If you pipe the output anywhere, remember $? after a pipe belongs to the last stage, not the script. Use pipefail or capture first.
$ python3 stale_procs.py --min-hours 100000
OK: no processes running deleted binaries
$ echo $?
0
# crontab: silent when clean, mail only when stale
30 6 * * 1 /usr/bin/python3 /opt/scripts/stale_procs.py --include-libs --min-hours 24Pair it with the rest of your post-patch hygiene: I use it alongside the OSV.dev unpatched-CVE report (package-to-CVE on the disk side) and the cron and systemd timer audit (scheduler side). The /proc/PID/maps trick generalizes too: same technique found the log files logrotate missed because a daemon kept an unlinked inode open. Reboot eventually anyway, but for services you cannot restart on a whim, this tells you whether last month’s security update is actually loaded.