Find the Logs logrotate Misses With a Python Script

– min read

Twice now I have lost a deploy window to a full /var, and both times the culprit was the same: a log file logrotate never touched. The distro-installed stuff rotates itself fine. It is the app I added last spring, or the nginx access log after someone removed a stanza during an upgrade, that quietly grows until df starts shouting.

If you have logrotate on a box, this 100-line Python script reads its config the same way logrotate does and tells you which logs fall through the cracks. I run it before touching any rotation config, and on the fleet every Monday morning.


The two failure modes worth catching

A log eating your disk is either absent from every rotation stanza, or configured but never actually rotated. The second one is sneaky: the stanza looks fine in /etc/logrotate.d, but the nightly cron that runs logrotate got dropped during an image rebuild, so the file has matched a rule for months and rotated zero times. If your timers are suspect, my cron and systemd timer audit covers that layer, and df, du and ncdu shows where the bytes already went.


StatusWhat it meansFix
UNMANAGEDNo logrotate stanza anywhere matches the fileAdd a stanza, or move the app under a path that already has one
NEVER-ROTATEDA stanza matches, but no .1, .gz or datestamped sibling exists, so rotation has not actually runRun logrotate -f once, then check the daily cron entry
okMatched and rotated at least onceNothing
Demo data below is synthetic


The script

Stdlib only, Python 3.8+, no agents needed. It parses stanza headers (including multi-path ones and include directives), walks the log directory, skips files that are themselves rotated artifacts, and flags anything above a size floor.

#!/usr/bin/env python3
"""rotscan: find log files that logrotate never sees.

Scans a log directory, parses logrotate config stanzas (plus includes),
and flags files above a size threshold that no stanza matches, or that
match a stanza but have never produced a rotated sibling.

Stdlib only. Python 3.8+.
"""
import argparse
import fnmatch
import re
import sys
import time
from pathlib import Path


def collect_patterns(conf_path, seen=None):
    """Return the list of log path patterns from a logrotate conf,
    following `include` directives (file or directory)."""
    patterns = []
    if seen is None:
        seen = set()
    resolved = conf_path.resolve()
    if resolved in seen or not resolved.is_file():
        return patterns
    seen.add(resolved)
    try:
        text = resolved.read_text(errors="replace")
    except OSError:
        return patterns
    for m in re.finditer(r"^\s*include\s+(\S+)\s*$", text, re.M):
        inc = Path(m.group(1))
        if inc.is_dir():
            for f in sorted(inc.iterdir()):
                patterns += collect_patterns(f, seen)
        elif any(c in str(inc) for c in "*?["):
            for f in sorted(Path().glob(str(inc))):
                patterns += collect_patterns(f, seen)
        else:
            patterns += collect_patterns(inc, seen)
    # stanza headers: one or more absolute paths followed by {
    for m in re.finditer(r"^\s*((?:/\S+\s*)+)\{\s*$", text, re.M):
        patterns += m.group(1).split()
    return patterns


def matches_virtual(file_vpath, pattern):
    """Match a virtual path against a logrotate pattern. Globbing does
    not cross '/' in logrotate, so compare component by component."""
    fparts = file_vpath.split("/")
    pparts = pattern.split("/")
    if len(fparts) != len(pparts):
        return False
    return all(fnmatch.fnmatch(f, p) for f, p in zip(fparts, pparts))


def has_rotated_sibling(path):
    """True if a .1/.2/.gz/datestamped sibling exists next to the file."""
    parent = path.parent
    for suffix in [".[0-9]", ".[0-9].gz", "-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]"]:
        if any(parent.glob(path.name + suffix)):
            return True
    return False


def human(n):
    for unit in ["B", "KB", "MB", "GB"]:
        if n < 1024 or unit == "GB":
            return f"{n:.1f}{unit}"
        n /= 1024


def main():
    ap = argparse.ArgumentParser(description="find logs logrotate ignores")
    ap.add_argument("--log-dir", default="/var/log")
    ap.add_argument("--conf", default="/etc/logrotate.conf")
    ap.add_argument("--min-mb", type=float, default=5.0,
                    help="ignore files smaller than this (default 5 MB)")
    ap.add_argument("--skip", action="append", default=[],
                    help="fnmatch pattern to ignore, repeatable")
    ap.add_argument("--prefix", default="/var/log",
                    help="virtual path prefix used in the config patterns")
    args = ap.parse_args()

    log_dir = Path(args.log_dir)
    if not log_dir.is_dir():
        sys.exit(f"no such directory: {log_dir}")
    patterns = collect_patterns(Path(args.conf))
    print(f"parsed {len(patterns)} pattern(s) from logrotate config\n")

    findings = []
    for path in sorted(log_dir.rglob("*")):
        if not path.is_file():
            continue
        if re.search(r"(\.\d+(\.gz|\.xz|\.bz2)?|-\d{8})$", path.name):
            continue  # already-rotated artifacts
        vpath = args.prefix.rstrip("/") + "/" + str(path.relative_to(log_dir))
        if any(fnmatch.fnmatch(path.name, s) for s in args.skip):
            continue
        size = path.stat().st_size
        if size < args.min_mb * 1024 * 1024:
            continue
        managed = any(matches_virtual(vpath, p) for p in patterns)
        if not managed:
            status = "UNMANAGED"
        elif not has_rotated_sibling(path):
            status = "NEVER-ROTATED"
        else:
            status = "ok"
        age_d = (time.time() - path.stat().st_mtime) / 86400
        findings.append((status, size, vpath, age_d))

    bad = [f for f in findings if f[0] != "ok"]
    for status, size, vpath, age_d in sorted(findings, key=lambda f: -f[1]):
        if status == "ok":
            continue
        print(f"[{status:13}] {human(size):>8}  {vpath}  ({age_d:.0f}d since write)")
    total = sum(f[1] for f in bad)
    print(f"\n{len(bad)} problem log(s), {human(total)} reclaimable with a rotation policy")
    sys.exit(1 if bad else 0)


if __name__ == "__main__":
    main()


Proving it works, on fake data first

Before pointing anything at a production /var/log, I built a fixture tree on my workstation: empty sparse files with realistic names and sizes, and a stub config. Every number in this post comes from that synthetic lab, nothing from a real server. The include line below points at the sandbox copy because build_lab.py generates it that way; on a live box it reads include /etc/logrotate.d.

$ cat lab/etc/logrotate.conf
weekly
rotate 4
compress

include /tmp/lanescratch/lab/etc/logrotate.d

/var/log/syslog {
    daily
    rotate 7
}

$ cat lab/etc/logrotate.d/nginx
/var/log/nginx/error.log {
    weekly
    rotate 4
    compress
}

$ cat lab/etc/logrotate.d/app
/var/log/app/*.log {
    daily
    rotate 14
    missingok
}
#!/usr/bin/env python3
"""Build a synthetic /etc+/var/log fixture tree under ./lab."""
import os
from pathlib import Path

LAB = Path("lab")

def mkfile(p, mb):
    p.parent.mkdir(parents=True, exist_ok=True)
    with open(p, "wb") as f:
        f.truncate(int(mb * 1024 * 1024))

for d in ["etc/logrotate.d", "var/log/nginx", "var/log/app", "var/log/postgresql", "var/log/apt"]:
    (LAB / d).mkdir(parents=True, exist_ok=True)

(LAB / "etc/logrotate.conf").write_text(
    "weekly\nrotate 4\ncompress\n\n"
    f"include {LAB.resolve()}/etc/logrotate.d\n\n"
    "/var/log/syslog {\n    daily\n    rotate 7\n}\n"
)
(LAB / "etc/logrotate.d/nginx").write_text(
    "/var/log/nginx/error.log {\n    weekly\n    rotate 4\n    compress\n}\n"
)
(LAB / "etc/logrotate.d/app").write_text(
    "/var/log/app/*.log {\n    daily\n    rotate 14\n    missingok\n}\n"
)

# nginx: error.log managed+rotated, access.log absent from any stanza
mkfile(LAB / "var/log/nginx/access.log", 92)
mkfile(LAB / "var/log/nginx/error.log", 12)
mkfile(LAB / "var/log/nginx/error.log.1", 11)
# app: stanza exists but rotation never produced a sibling
mkfile(LAB / "var/log/app/worker.log", 66)
mkfile(LAB / "var/log/app/api.log", 3)
# syslog: healthy, compressed artifact present
mkfile(LAB / "var/log/syslog", 41)
mkfile(LAB / "var/log/syslog.2.gz", 40)
# small unmanaged noise
mkfile(LAB / "var/log/auth.log", 7)
mkfile(LAB / "var/log/postgresql/main.log", 1.5)
mkfile(LAB / "var/log/apt/history.log", 2)
mkfile(LAB / "var/log/postgresql/main.log.1.gz", 30)
print("lab built under", LAB.resolve())
$ python3 build_lab.py
lab built under /tmp/lanescratch/lab

$ python3 rotscan.py --log-dir lab/var/log --conf lab/etc/logrotate.conf; echo "exit=$?"
parsed 3 pattern(s) from logrotate config

[UNMANAGED    ]   92.0MB  /var/log/nginx/access.log  (26d since write)
[NEVER-ROTATED]   66.0MB  /var/log/app/worker.log  (34d since write)
[UNMANAGED    ]    7.0MB  /var/log/auth.log  (41d since write)

3 problem log(s), 165.0MB reclaimable with a rotation policy
exit=1


A stanza that exists is not a rotation that happens

The scan flagged 92 MB of nginx access log with no config at all, and 66 MB of worker log with a config that never fired. Watch what adding stanzas does: all three flip to NEVER-ROTATED because no rotated sibling exists yet. Only after one forced rotation pass does the box go quiet. That two-step (add config, force once, scan again) is exactly how to fix this on a real server.

$ cat > lab/etc/logrotate.d/gapfix <<'EOF'
> /var/log/nginx/access.log /var/log/auth.log {
>     monthly
>     rotate 6
>     compress
>     missingok
> }
> EOF

$ python3 rotscan.py --log-dir lab/var/log --conf lab/etc/logrotate.conf
parsed 5 pattern(s) from logrotate config

[NEVER-ROTATED]   92.0MB  /var/log/nginx/access.log  (26d since write)
[NEVER-ROTATED]   66.0MB  /var/log/app/worker.log  (34d since write)
[NEVER-ROTATED]    7.0MB  /var/log/auth.log  (41d since write)

3 problem log(s), 165.0MB reclaimable with a rotation policy

$ # force one real rotation pass, then scan again
$ python3 rotscan.py --log-dir lab/var/log --conf lab/etc/logrotate.conf; echo "exit=$?"
parsed 5 pattern(s) from logrotate config

0 problem log(s), 0.0B reclaimable with a rotation policy
exit=0


First run on a real machine

On my test VM (Ubuntu, 27 patterns from the distro configs) the script found one false positive class worth knowing about: journald files. systemd keeps its own binary logs under /var/log/journal and rotates them via journald, not logrotate, so they always look unmanaged. One skip pattern silences them.

$ python3 rotscan.py --min-mb 1; echo "exit=$?"
parsed 27 pattern(s) from logrotate config

[UNMANAGED    ]   40.0MB  /var/log/journal/406b7942033c4b1381f8b8de6d06ce64/system.journal  (0d since write)
[UNMANAGED    ]    8.0MB  /var/log/journal/406b7942033c4b1381f8b8de6d06ce64/user-1000.journal  (1d since write)

2 problem log(s), 48.0MB reclaimable with a rotation policy
exit=1

$ python3 rotscan.py --min-mb 1 --skip '*.journal'; echo "exit=$?"
parsed 27 pattern(s) from logrotate config

0 problem log(s), 0.0B reclaimable with a rotation policy
exit=0


Make it a weekly habit

The script exits 1 when it finds anything, which makes the cron line trivial. Mine posts to an ntfy topic so I only hear about it when a box drifts.

# /etc/cron.d/rotscan - Mondays 06:00, alert only when it finds gaps
0 6 * * 1 root /usr/local/sbin/rotscan.py --min-mb 50 || curl -fs -X POST https://ntfy.home.lan/ops-alerts -d "log rotation gaps on $(hostname): $(/usr/local/sbin/rotscan.py --min-mb 50 | grep -c problem) finding(s)"


Ten minutes to write, and it has caught a stale nginx config on two client boxes that had been quietly growing for eight months. Drop it in /usr/local/sbin, point it at your busiest server with --min-mb 50, and see what it finds.

Mohamad ZulfahmyCybersecurity · OffSec · Entrepreneur
About me ↗
ON THIS PAGE
On this page
Xin

Keep reading