Audit journald disk usage with Python

– min read

A while back my SSH box reported “Archived and active journals take up 98.5M” and that was it. Not one command tells you which service wrote those 98 megabytes. So I wrote journal_hogs.py: it streams the journal once, attributes every byte to its unit, ranks the talkers, and exits non-zero when one of them is spamming.

Official docs: systemd.io · systemd-journald(8) man page · GitHub (systemd/systemd)


Why –disk-usage is a dead end

journalctl --disk-usage prints one number for the whole journal. Journal files are compressed on disk, so even that number hides the real text volume. There is no built-in per-unit accounting: journalctl -u something counts entries for one unit you already suspect, which is useless when you are asking who to suspect. What we do have is journalctl -o json: every entry as one JSON line, with a _SYSTEMD_UNIT field on almost all of them. Stream that, sum MESSAGE byte lengths per unit, divide by the wall-clock window the entries span, done.


The script

Pure standard library, Python 3.8 and up, works over rotated archived journals because journalctl merges them for you. The --json flag and the exit codes are there so you can drop it into cron or a monitoring check later.

#!/usr/bin/env python3
"""journal_hogs.py - attribute journald growth to systemd units.

Streams journalctl JSON output, sums payload bytes per unit, ranks the
noisiest writers, flags hogs by bytes/sec, and prints vacuum advice.
Read-only unless --vacuum is passed.
"""
import argparse
import base64
import collections
import json
import re
import subprocess
import sys
from datetime import datetime


def entry_bytes(entry):
    msg = entry.get("MESSAGE", b"")
    if isinstance(msg, str):
        m = re.fullmatch(r"y([A-Za-z0-9+/=]*)", msg)
        if m:
            msg = base64.b64decode(m.group(1))
        else:
            msg = msg.encode("utf-8", "replace")
    return len(msg)


def unit_of(entry):
    return (entry.get("_SYSTEMD_UNIT")
            or entry.get("SYSLOG_IDENTIFIER")
            or entry.get("_COMM")
            or "[no unit]")


def stream(since, until, unit):
    cmd = ["journalctl", "-o", "json", "--no-pager", "-q"]
    if since:
        cmd += ["--since", since]
    if until:
        cmd += ["--until", until]
    if unit:
        cmd += ["_SYSTEMD_UNIT=" + unit]
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                            stderr=subprocess.DEVNULL, text=True)
    for line in proc.stdout:
        line = line.strip()
        if line:
            yield json.loads(line)
    proc.wait()


def main():
    ap = argparse.ArgumentParser(description="rank journald writers per unit")
    ap.add_argument("--since", default="7 days ago")
    ap.add_argument("--until", default=None)
    ap.add_argument("--unit", default=None, help="filter one unit")
    ap.add_argument("--top", type=int, default=15)
    ap.add_argument("--hog-mbps", type=float, default=0.05,
                    help="bytes/sec threshold to flag a unit (default 0.05)")
    ap.add_argument("--json", action="store_true", help="machine output")
    args = ap.parse_args()

    t0 = datetime.now()
    stats = collections.defaultdict(
        lambda: {"bytes": 0, "entries": 0, "samples": {}})
    total_bytes = 0
    total_entries = 0
    min_ts = None
    max_ts = None

    for entry in stream(args.since, args.until, args.unit):
        u = unit_of(entry)
        b = entry_bytes(entry)
        stats[u]["bytes"] += b
        stats[u]["entries"] += 1
        if len(stats[u]["samples"]) < 3:
            raw = entry.get("MESSAGE", "")
            if isinstance(raw, str) and raw.startswith("y") and not raw[1:2].isalnum():
                raw = raw[:80]
            stats[u]["samples"].setdefault(
                raw.encode("utf-8", "replace")[:120].decode("utf-8", "replace"), 0)
        total_bytes += b
        total_entries += 1
        ts = entry.get("__REALTIME_TIMESTAMP")
        if ts:
            ts = int(ts) / 1_000_000
            min_ts = ts if min_ts is None else min(min_ts, ts)
            max_ts = ts if max_ts is None else max(max_ts, ts)

    window = max((max_ts or 0) - (min_ts or 0), 1.0)

    ranked = sorted(stats.items(), key=lambda kv: kv[1]["bytes"], reverse=True)
    hogs = [(u, s) for u, s in ranked
            if s["bytes"] / window > args.hog_mbps]

    disk = subprocess.run(["journalctl", "--disk-usage"],
                          capture_output=True, text=True).stdout.strip()

    if args.json:
        out = {"window_seconds": round(window),
               "total_bytes": total_bytes, "total_entries": total_entries,
               "disk": disk,
               "units": [{"unit": u, "bytes": s["bytes"],
                          "entries": s["entries"],
                          "bps": round(s["bytes"] / window, 4),
                          "pct": round(100 * s["bytes"] / max(total_bytes, 1), 2)}
                         for u, s in ranked]}
        print(json.dumps(out, indent=2))
        sys.exit(1 if hogs else 0)

    print(disk)
    print(f"window: {window/3600:.1f} h | entries: {total_entries:,} | "
          f"payload: {total_bytes/1_048_576:.1f} MiB | "
          f"elapsed: {(datetime.now()-t0).total_seconds():.1f}s")
    print()
    print(f"{'UNIT':<38}{'ENTRIES':>10}{'SIZE':>10}{'%':>7}{'B/s':>8}")
    for u, s in ranked[:args.top]:
        pct = 100 * s["bytes"] / max(total_bytes, 1)
        size = (f"{s['bytes']/1_048_576:.1f}M" if s["bytes"] >= 1_048_576
                else f"{s['bytes']/1024:.0f}K")
        print(f"{u:<38}{s['entries']:>10,}{size:>10}{pct:>6.1f}%"
              f"{s['bytes']/window:>8.3f}")
    print()
    if hogs:
        print("FLAGGED (above --hog-mbps=" + str(args.hog_mbps) + "):")
        for u, s in hogs:
            sample = next(iter(s["samples"]), "")
            print(f"  {u}  [{sample[:70]}]")
        print()
        print("fix: cap the unit with a drop-in, then shrink the journal:")
        print("  systemctl edit    # add RateLimitIntervalSec / LogLevel")
        print("  journalctl --vacuum-size=200M")
        sys.exit(1)
    print("no unit exceeded the rate threshold")
    sys.exit(0)


if __name__ == "__main__":
    main()


The first real run found a mess

This is my actual runner (Ubuntu 26.04, systemd 259), a quiet box that should log almost nothing:

python3 journal_hogs.py --since "7 days ago" --top 12
Archived and active journals take up 90M in the file system.
window: 165.6 h | entries: 176,299 | payload: 33.4 MiB | elapsed: 9.7s

UNIT                                     ENTRIES      SIZE      %     B/s
rsyslog.service                          171,500     33.0M  99.0%  58.067
log-flood-demo.service                     2,382      113K   0.3%   0.194
init.scope                                   921       78K   0.2%   0.134
ntfy.service                                 146       37K   0.1%   0.063
kernel                                       345       30K   0.1%   0.052
containerd.service                            78       15K   0.0%   0.026
docker.service                                41       10K   0.0%   0.017
networkd-dispatcher.service                   92        9K   0.0%   0.015
tailscaled.service                            76        7K   0.0%   0.011
udisks2.service                              104        7K   0.0%   0.011
NetworkManager.service                        54        5K   0.0%   0.009
systemd-coredump                              54        5K   0.0%   0.009

FLAGGED (above --hog-mbps=0.05):
  rsyslog.service  [rsyslogd: file '/var/log/syslog'[7] write error - see https://www.rsys]
  log-flood-demo.service  [synthetic demo line 1 worker=pool-3 queue=jobs]
  init.scope  [var-lib-containerd-tmpmounts-containerd\x2dmount2661837370.mount: Deac]
  ntfy.service  [INFO Server stats (emails_received=0, emails_received_failure=0, email]
  kernel  [br-d3d8431f83a5: port 1(veth92ad328) entered disabled state]

fix: cap the unit with a drop-in, then shrink the journal:
  systemctl edit    # add RateLimitIntervalSec / LogLevel
  journalctl --vacuum-size=200M
exit=1

That 99.0% on rsyslog was the finding. Drilling in with journalctl -u rsyslog.service -n 6 showed a write-error loop: file /var/log/syslog write error, OS error: No space left on device. The root disk had briefly hit 95% full, rsyslog could not write its action output, and it logged the failure pair over and over, about 171,500 entries in two days, straight into the journal. The loop ended by itself when space freed up, but nobody would have noticed without the byte ranking: nothing was crashing, so every health check stayed green.


Flow diagram of journal_hogs.py: journalctl JSON stream, per-unit attribution, bytes-per-second scoring, exit code and fixes


Seeing the flagging with synthetic noise

To test the hog-flagging path I generated synthetic traffic: a throwaway systemd-run unit writing 30,000 lines of made-up worker chatter. No real data involved, the pool-3 queue names are invented for the demo.

systemd-run --unit=log-flood-demo --collect bash -c 'for i in $(seq 1 30000); do echo "synthetic demo line $i worker=pool-3 queue=jobs"; done'
python3 journal_hogs.py --since "1 hour ago" --top 6
Archived and active journals take up 95.7M in the file system.
window: 1.0 h | entries: 22,600 | payload: 1.1 MiB | elapsed: 1.3s

UNIT                                     ENTRIES      SIZE      %     B/s
log-flood-demo.service                    22,382      1.1M  98.0% 307.531
ntfy.service                                  60       15K   1.4%   4.327
init.scope                                    92        5K   0.5%   1.532
cron.service                                   9        1K   0.1%   0.187
rl-test.service                               40        0K   0.0%   0.142
udisks2.service                                4        0K   0.0%   0.072

FLAGGED (above --hog-mbps=0.05):
  log-flood-demo.service  [synthetic demo line 1 worker=pool-3 queue=jobs]
  ntfy.service  [INFO Server stats (emails_received=0, emails_received_failure=0, email]
  init.scope  [Starting fwupd-refresh.service - Refresh fwupd metadata and update mot]
  cron.service  [pam_unix(cron:session): session opened for user root(uid=0) by root(ui]
  rl-test.service  [flood line 1]
  udisks2.service  [Error statting /dev/dm-1\040(deleted): No such file or directory]

fix: cap the unit with a drop-in, then shrink the journal:
  systemctl edit    # add RateLimitIntervalSec / LogLevel
  journalctl --vacuum-size=200M
exit=1

22,382 entries, 98% of the hour of bytes, flagged at 307 bytes per second. That is the shape of the output you use to aim the fix.


Stopping the noise

systemd gained per-unit journal rate limits, and I verified them live on 259: the same 3,000-line flood under a burst cap keeps 85 entries, the rest is dropped.

systemd-run -p LogRateLimitIntervalSec=10s -p LogRateLimitBurst=20 --unit=rl-test bash -c 'for i in $(seq 1 3000); do echo "flood line $i"; done'

lines written: 3000
journal entries kept:
$ journalctl -u rl-test --no-pager | wc -l
85

For a real service the same two keys go in a drop-in. One gotcha I hit while testing: on systemd 259 the keys must sit in the [Service] section. I put them under [Unit] first and systemctl show kept reporting the inherited default of 0, silently. In systemctl edit myservice.service:

[Service]
LogRateLimitIntervalSec=30s
LogRateLimitBurst=500


Keeping the journal small forever

# vacuum what is already there
journalctl --vacuum-size=200M

# or set the ceiling so journald self-trims from now on
# /etc/systemd/journald.conf -> [Journal]
SystemMaxUse=500M

If your box still keeps plain text logs next to the journal, find the logs logrotate misses first, this script only sees what goes through journald. And before adding more timers, audit your cron and systemd timer hygiene so the noise you chase is not your own 5-minute check-ins.


Cheat sheet of journald commands to rank log writers, cap a unit with rate limits, set SystemMaxUse and vacuum the journal


The cron hook

*/60 * * * * root /usr/local/bin/journal_hogs.py --since "6 hours ago" --json > /tmp/journal-report.json 2>> /tmp/journal-report.err || /usr/local/bin/alert.sh "$(jq -r '[.units[] | select(.bps > 0.05) | .unit] | join(", ")' /tmp/journal-report.json)"

Exit code 1 means at least one unit crossed --hog-mbps, so the cron line alerts with just the unit names. For a full resource-cap picture beyond logs, the systemd resource limits per service write-up covers the CPU and memory side of the same drop-in workflow.


One streaming pass, no parsing of human output, and you always end up with the number that --disk-usage should have carried in the first place: who to tell to shut up.

Mohamad ZulfahmyCybersecurity · OffSec · Entrepreneur
About me ↗
ON THIS PAGE
On this page