Merge IP Allowlists With a Python Script

– min read

Four teams, four CSVs, one firewall allowlist. Every quarter the merged file grows entries that three people added without talking to each other, and the person doing the merge (me, this time) ends up eyeballing CIDRs at midnight.


So I wrote allowlist_merge.py: a stdlib-only Python script that takes any number of team CSVs, dedupes them, flags ranges claimed by more than one owner, finds entries already swallowed by a wider subnet, and emits a clean nftables set. The whole thing runs in under a second and the exit code is the number of findings, so cron can alarm on a messy merge.


Source links

The two modules doing all the work: Python ipaddress docs and Python csv docs. If your firewall side is nftables, my earlier walkthrough is here.


The input files

Each team submits a CSV with network,owner,note columns. Three of my synthetic demo files look like this. All addresses are RFC 5737 documentation ranges, by the way, nothing here is a real allowlist.


network,owner,note
203.0.113.10,itops,jenkins CI
203.0.113.0/27,itops,office VPN pool
203.0.113.44,itops,vpn exit
198.51.100.7,itops,monitoring probe
192.0.2.10,itops,old bastion (keep)


network,owner,note
203.0.113.0/25,office,HQ subnet
203.0.113.44,office,same vpn exit
203.0.113.130,office,reception wi-fi


The vendors file is where it gets fun: one entry duplicates the bastion, one is covered by their own CDN range, and somebody pasted a hostname into the IP column.

network,owner,note
192.0.2.0/28,vendors,partner B2B range
192.0.2.10,vendors,their api egress
203.0.113.200/29,vendors,saas scanner
198.51.100.0/24,vendors,cdn nodes
webhook.vendor.example,vendors,"oops, hostname"


The script

One detail worth knowing up front: ipaddress.ip_network() with strict=False silently normalises a host address like 203.0.113.44 that a team pasted as a bare IP into whatever subnet it belongs to, which is exactly the bug class we are hunting, so the script never passes strict=False for single hosts and relies on the interface-style normalisation shown below to fold /32 forms consistently.


#!/usr/bin/env python3
"""Merge team IP allowlist CSVs, dedupe, and flag redundancy.

Each CSV needs columns: network, owner, note.
Exit code = number of findings (conflicts + redundant coverages).
"""
import argparse, csv, ipaddress, sys
from collections import defaultdict
from pathlib import Path


def normalise(raw):
    net = ipaddress.ip_network(raw.strip(), strict=False)
    if net.prefixlen == net.max_prefixlen:      # /32 on IPv4 -> plain address
        net = ipaddress.ip_network(f"{net.network_address}/32")
    return net


def load(paths):
    entries, bad = [], []
    for path in paths:
        with open(path, newline="") as fh:
            for lineno, row in enumerate(csv.DictReader(fh), start=2):
                try:
                    net = normalise(row["network"])
                except ValueError:
                    bad.append((path.name, lineno, row["network"]))
                    continue
                entries.append((net, row["owner"].strip(),
                                row["note"].strip(), Path(path).name, lineno))
    return entries, bad


def main():
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("csvs", nargs="+", type=Path, help="allowlist CSV files")
    ap.add_argument("--nft-file", type=Path, help="also write an nftables set")
    args = ap.parse_args()

    entries, bad = load(args.csvs)

    # group identical networks
    by_net = defaultdict(list)
    for net, owner, note, fname, lineno in entries:
        by_net[net].append((owner, note, fname, lineno))

    nets = sorted(by_net, key=lambda n: (n.network_address.packed, n.prefixlen))

    # exact duplicates across owners -> conflict report
    conflicts = {n: rows for n, rows in by_net.items()
                 if len({r[0] for r in rows}) > 1}

    # redundant coverage: entry fully inside a broader entry owned elsewhere
    redundant = []
    for n in nets:
        for b in nets:
            if b.version != n.version or b.prefixlen >= n.prefixlen:
                continue
            if n.subnet_of(b):
                owners_b = sorted({r[0] for r in by_net[b]})
                redundant.append((n, b, owners_b))
                break

    print(f"parsed {len(entries)} entries from {len(args.csvs)} files")
    print(f"unique networks after dedupe: {len(nets)}")

    if bad:
        print(f"skipped {len(bad)} unparseable rows:")
        for f, ln, val in bad:
            print(f"  {f}:{ln} {val!r} is not an IP or CIDR")

    print(f"\nconflicts (same range claimed by >1 owner): {len(conflicts)}")
    for n, rows in sorted(conflicts.items(), key=lambda kv: str(kv[0])):
        who = sorted({(r[0], r[2], r[3]) for r in rows})
        print(f"  {n}  claimed by " + ", ".join(f"{o} ({f}:{l})" for o, f, l in who))

    print(f"\nredundant entries (already covered by a wider range): {len(redundant)}")
    for n, b, owners in redundant:
        print(f"  {n} is inside {b} (owner: {','.join(sorted(owners))}) -> drop candidate")

    keep = [n for n in nets if n not in {r[0] for r in redundant}]
    print(f"\nfinal list: {len(keep)} entries\n")
    for n in keep:
        owners = "/".join(sorted({r[0] for r in by_net[n]}))
        print(f"  {str(n.network_address):>15}  prefix /{n.prefixlen:<3} owner {owners}")

    if args.nft_file:
        lines = ["table inet fw {", "  set allowlist {",
                 "    type ipv4_addr", "    flags interval", "    elements = {"]
        body = [f"        {n}" for n in keep]
        lines += [l + ("," if i < len(body) - 1 else "") for i, l in enumerate(body)]
        lines += ["    }", "  }", "}", ""]
        args.nft_file.write_text("\n".join(lines))
        print(f"wrote nftables set to {args.nft_file}")

    findings = len(conflicts) + len(redundant)
    print(f"\n{findings} finding(s)")
    sys.exit(min(findings, 250))


if __name__ == "__main__":
    main()


Running it

$ cd /etc/firewall/allowlists
$ python3 allowlist_merge.py data/*.csv --nft-file out.nft
echo $?


On the demo data the report looks like this. This is the real captured stdout, not a mockup.


parsed 12 entries from 3 files
unique networks after dedupe: 10
skipped 1 unparseable rows:
  vendors.csv:6 'webhook.vendor.example' is not an IP or CIDR

conflicts (same range claimed by >1 owner): 2
  192.0.2.10/32  claimed by itops (itops.csv:6), vendors (vendors.csv:3)
  203.0.113.44/32  claimed by itops (itops.csv:4), office (office.csv:3)

redundant entries (already covered by a wider range): 5
  192.0.2.10/32 is inside 192.0.2.0/28 (owner: vendors) -> drop candidate
  198.51.100.7/32 is inside 198.51.100.0/24 (owner: vendors) -> drop candidate
  203.0.113.0/27 is inside 203.0.113.0/25 (owner: office) -> drop candidate
  203.0.113.10/32 is inside 203.0.113.0/25 (owner: office) -> drop candidate
  203.0.113.44/32 is inside 203.0.113.0/25 (owner: office) -> drop candidate

final list: 5 entries

        192.0.2.0  prefix /28  owner vendors
     198.51.100.0  prefix /24  owner vendors
      203.0.113.0  prefix /25  owner office
    203.0.113.130  prefix /32  owner office
    203.0.113.200  prefix /29  owner vendors
wrote nftables set to out.nft

7 finding(s)


Terminal output showing conflicts and redundant entries found by allowlist_merge.py


Twelve entries in, ten unique networks, and seven findings: two conflicts where two teams both claim the same address, plus five drop candidates that a wider range already covers. The 192.0.2.10 bastion that IT swore was special? The vendor route already lets them in, and the note column tells you who to argue with.


The nftables output

The generated set is a single --nft-file away, and you can validate it before loading. Same five kept entries as the report:


table inet fw {
  set allowlist {
    type ipv4_addr
    flags interval
    elements = {
        192.0.2.0/28,
        198.51.100.0/24,
        203.0.113.0/25,
        203.0.113.130/32,
        203.0.113.200/29
    }
  }
}


$ nft --check -f out.nft && echo ok
ok


Final deduplicated allowlist and nftables set generated by the script


One bug I hit while verifying

My first version wrote a broken nft file. The elements = { block never got its closing brace because I reused the set and table closers as one list, so the output was two braces deep for a three-level structure. nft --check caught it immediately, which is why that validation line is in the post. If you write this kind of generator, always pipe the output through the tool that consumes it before you trust the report.


Making it part of the routine

The exit code equals the finding count, so a nightly cron entry that mails you only on a non-zero result does the alerting for free. That pairs nicely with the cron and systemd timer hygiene audit I wrote about, and if your allowlists are actually SSH-based, the sshd_config hardening audit catches the other half of this problem.


Run it before your next firewall change. The twenty seconds it spends finding duplicates beats the twenty minutes of Slack archaeology.

Mohamad ZulfahmyCybersecurity · OffSec · Entrepreneur
About me ↗
ON THIS PAGE
On this page