Audit Listening Sockets From /proc

– min read

Something is answering on port 45321 of your web server and nobody knows who. You log in, run ss -tuln, and on that particular box it is not installed: slimmed container image, no iproute2. This script is the answer I wrote for that moment: a full listener audit that only needs Python and a kernel that exposes /proc.

Docs for the data sources: the proc(5) manual and iproute2 for the tool we are replacing.

Flow diagram of the listen_audit.py pipeline from /proc/net socket tables to systemd unit mapping
How the audit flows from raw kernel files to a named service. Author’s own diagram.


Why parse /proc instead of running ss

Two reasons. Availability: /proc/net/tcp exists on every Linux, even the 8 MB images. Trust: every userland tool reads sockets through libc, and the classic rootkit move is hooking libc so tools like netstat cheerfully hide the backdoor. Walking /proc/net/* plus the /proc/PID/fd symlinks talks to the kernel directly. Same reason I like auditing deleted binaries via /proc: you are reading what the kernel actually says, not what a wrapper says.


What the kernel hands you

Each of /proc/net/tcp, tcp6, udp and udp6 is a text table, one row per socket. A listening row looks like this (a demo socket from my sandbox, trimmed):

   7: B80D0120000000000000000001000000:21E3 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 8440798 1 ffff8ae052512800 100 0 0 10 0

Field 2 is local address and port, field 3 the remote pair, field 4 the state (0A = LISTEN), field 10 the socket inode. Addresses are hex, and that hex is where both bugs in my first version lived. More on that below.


The script

One file, stdlib only. Run it as root so it can read other users’ fd links. Table to stdout, one-line summary to stderr, exit code 1 when anything listens beyond loopback, which makes it cron-ready as-is. --json gives machines the same data, --all includes loopback.

#!/usr/bin/env python3
"""listen_audit.py -- inventory every listening socket from /proc only.
No ss, no netstat, no lsof. Useful on stripped boxes where those tools
are missing or suspect (an attacker's rootkit often hooks libc first).
"""
import sys, os, glob, argparse, ipaddress

TCP_LISTEN = "0A"

def parse_addr(hexaddr):
    addr_hex, port_hex = hexaddr.split(":")
    port = int(port_hex, 16)
    if len(addr_hex) == 8:                        # IPv4: one LE 32-bit word
        raw = bytes.fromhex(addr_hex)[::-1]
    else:                                         # IPv6: FOUR LE 32-bit words
        raw = b"".join(bytes.fromhex(addr_hex[i:i+8])[::-1]
                       for i in range(0, 32, 8))
    return ipaddress.ip_address(raw), port

def socket_inodes():
    """Read the kernel's socket tables, yield (proto, local_ip, port, inode)."""
    for path, proto in (("/proc/net/tcp", "tcp"), ("/proc/net/tcp6", "tcp6"),
                        ("/proc/net/udp", "udp"), ("/proc/net/udp6", "udp6")):
        try:
            lines = open(path).read().splitlines()[1:]
        except OSError:
            continue
        for line in lines:
            f = line.split()
            if proto.startswith("tcp") and f[3] != TCP_LISTEN:
                continue
            if proto.startswith("udp"):
                # "listening" UDP == unconnected: remote all-zeros.
                # The remote PORT is always 4 hex digits, but the remote
                # ADDRESS is 8 hex digits on v4 and 32 on v6 -- a literal
                # string match breaks on one of the two families. Compare
                # the parsed values instead.
                rip, rport = parse_addr(f[2])
                if rport != 0 or int(rip) != 0:
                    continue
            ip, port = parse_addr(f[1])
            yield proto, ip, port, f[9]

def inode_to_pid():
    """Walk every /proc//fd symlink to map socket inodes to owners."""
    owners = {}
    for pid_dir in glob.glob("/proc/[0-9]*/fd"):
        pid = pid_dir.split("/")[2]
        try:
            for fd in os.scandir(pid_dir):
                try:
                    link = os.readlink(fd.path)
                except OSError:
                    continue
                if link.startswith("socket:["):
                    owners[link[8:-1]] = pid
        except OSError:
            pass          # process exited mid-scan, or permission denied
    return owners

def pid_info(pid):
    try:
        comm = open(f"/proc/{pid}/comm").read().strip()
    except OSError:
        comm = "?"
    unit = "-"
    try:
        for cg in open(f"/proc/{pid}/cgroup"):
            parts = cg.strip().split(":")[-1]
            for seg in reversed(parts.split("/")):
                if seg.endswith((".service", ".socket", ".timer")):
                    unit = seg
                    break
    except OSError:
        pass
    return comm, unit

def scope(ip):
    if ip.is_loopback:
        return "local"
    if ip == ipaddress.ip_address("0.0.0.0") or ip == ipaddress.ip_address("::"):
        return "WILDCARD"
    if ip.version == 6 and getattr(ip, "ipv4_mapped", None):
        return scope(ip.ipv4_mapped)
    return "public-iface"

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--all", action="store_true", help="show loopback too")
    ap.add_argument("--json", action="store_true")
    args = ap.parse_args()

    owners = inode_to_pid()
    rows = []
    for proto, ip, port, inode in socket_inodes():
        pid = owners.get(inode, "?")
        comm, unit = ("?", "?") if pid == "?" else pid_info(pid)
        s = scope(ip)
        if s == "local" and not args.all:
            continue
        rows.append({"proto": proto, "addr": str(ip), "port": port,
                     "scope": s, "pid": pid, "comm": comm, "unit": unit})

    if args.json:
        import json
        print(json.dumps(rows, indent=2))
    else:
        hdr = f"{'PROTO':6}{'LOCAL ADDRESS':26}{'SCOPE':16}{'PID':8}{'PROCESS':16}UNIT"
        print(hdr); print("-" * len(hdr))
        for r in sorted(rows, key=lambda r: (r["scope"] != "WILDCARD", r["port"])):
            print(f"{r['proto']:6}{r['addr'] + ':' + str(r['port']):26}"
                  f"{r['scope']:16}{str(r['pid']):8}{r['comm']:16}{r['unit']}")
    risky = [r for r in rows if r["scope"] in ("WILDCARD", "public-iface")]
    # human summary goes to stderr so stdout stays pure JSON with --json
    print(f"{len(rows)} listeners, {len(risky)} reachable beyond loopback.",
          file=sys.stderr)
    sys.exit(1 if risky else 0)

if __name__ == "__main__":
    main()


Testing it: planted listeners

I validated on my monitoring runner, which already has real listeners (sshd, prometheus, grafana, rpcbind). On top of those I planted four demo sockets bound to the documentation ranges from RFC 5737 and RFC 3849, 203.0.113.10 and 2001:db8::1, plus one wildcard TCP and one loopback TCP. All the demo addresses and processes in this article are synthetic; nothing here was scanned from a live public interface:

ip addr add 203.0.113.10/32 dev lo
ip -6 addr add 2001:db8::1/128 dev lo
# one-liner each: two TCP, one UDP, two IPv6 listeners
python3 -c "import socket,time;s=socket.socket();s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1);s.bind(('0.0.0.0',33741));s.listen(5);time.sleep(9999)" &
python3 -c "import socket,time;s=socket.socket(socket.AF_INET6,socket.SOCK_DGRAM);s.bind(('203.0.113.10',4521));time.sleep(9999)" &

Then the audit. This is the real captured output, note the two planted public-iface rows and the WILDCARD rows sorted to the top:

28 listeners, 17 reachable beyond loopback.
PROTO LOCAL ADDRESS             SCOPE           PID     PROCESS         UNIT
----------------------------------------------------------------------------
tcp   0.0.0.0:22                WILDCARD        5832    sshd            ssh.service
tcp6  :::22                     WILDCARD        5832    sshd            ssh.service
tcp   0.0.0.0:111               WILDCARD        474796  rpcbind         rpcbind.service
tcp6  :::111                    WILDCARD        474796  rpcbind         rpcbind.service
udp   0.0.0.0:111               WILDCARD        474796  rpcbind         rpcbind.service
udp6  :::111                    WILDCARD        474796  rpcbind         rpcbind.service
tcp6  :::3000                   WILDCARD        236194  grafana         grafana-server.service
tcp6  :::9090                   WILDCARD        236147  prometheus      prometheus.service
tcp6  :::9100                   WILDCARD        234177  prometheus-node prometheus-node-exporter.service
tcp6  :::31337                  WILDCARD        565564  python3         -
tcp   0.0.0.0:33741             WILDCARD        565561  python3         -
tcp6  :::42615                  WILDCARD        475253  rpc.statd       rpc-statd.service
tcp   0.0.0.0:56257             WILDCARD        475253  rpc.statd       rpc-statd.service
udp6  :::58471                  WILDCARD        475253  rpc.statd       rpc-statd.service
udp   0.0.0.0:60306             WILDCARD        475253  rpc.statd       rpc-statd.service
tcp   127.0.0.54:53             local           5494    systemd-resolve systemd-resolved.service
tcp   127.0.0.53:53             local           5494    systemd-resolve systemd-resolved.service
udp   127.0.0.54:53             local           5494    systemd-resolve systemd-resolved.service
udp   127.0.0.53:53             local           5494    systemd-resolve systemd-resolved.service
udp   127.0.0.1:323             local           12593   chronyd         chrony.service
udp6  ::1:323                   local           12593   chronyd         chrony.service
udp   127.0.0.1:973             local           475253  rpc.statd       rpc-statd.service
tcp   127.0.0.1:3306            local           231004  mariadbd        mariadb.service
udp   203.0.113.10:4521         public-iface    565563  python3         -
tcp   127.0.0.1:4711            local           565562  python3         -
tcp   127.0.0.1:8080            local           232825  php             -
tcp6  2001:db8::1:8675          public-iface    565565  python3         -
tcp   127.0.0.1:42221           local           499414  containerd      containerd.service
Terminal output of listen_audit.py showing wildcard and public-iface listeners with owning processes
Audit output with the planted demo listeners. Author’s own sandbox capture.


Two bugs the sandbox caught

First version printed this for my 2001:db8::1 listener: 0:1::2001:db8:8675. Garbage. For IPv4 the kernel stores one little-endian 32-bit word, so reversing the whole byte string works. IPv6 is four little-endian words: reverse each 8-hex-digit group separately, not all 16 bytes at once. Every all-zeros :: row hid the bug until a real v6 address showed up.

# wrong for v6:
raw = bytes.fromhex(addr_hex)[::-1]
# right: reverse inside each 32-bit word
raw = b"".join(bytes.fromhex(addr_hex[i:i+8])[::-1] for i in range(0, 32, 8))

Second bug: UDP listeners simply never appeared. I was filtering rem_address != "00000000:00000000", copied from the v4 table. But UDP rows have no LISTEN state at all (they sit in 07), and the remote port is always 4 hex digits while the remote address is 8 digits on v4 and 32 on v6. One of my two literal string matches was destined to fail; comparing parsed values instead of strings fixed it. Lesson for any /proc parser: parse, do not string-match.


Making it an alarm

The exit code is the feature. A cron entry that only messages you when the listener set changes shape, i.e. anything new opens beyond loopback, is one line. For the report itself, --json plus jq does the filtering; jq, like the ss it often pairs with, is the one convenience you may want to install:

# who is on non-loopback ports right now
python3 listen_audit.py --json | jq -r '.[] | select(.scope != "local") | "\(.proto) \(.addr):\(.port) \(.comm) [\(.unit)]"'
# cron: alert only when something beyond loopback exists
*/30 * * * * /usr/bin/python3 /usr/local/bin/listen_audit.py >/dev/null 2>&1 || /usr/local/bin/notify.sh "listeners on $(hostname)"


The report is boring when nothing changed, which is exactly what you want from a hygiene script. If you run one on a fleet, the cgroup column makes the diff instantly actionable, you get a unit name instead of a PID. My earlier posts on checking cert expiry across a host list and summarizing fail2ban bans follow the same shape: parse kernel or log truth, print a table, exit nonzero when a human should look.

Mohamad ZulfahmyCybersecurity · OffSec · Entrepreneur
About me ↗
ON THIS PAGE
On this page