Four teams, four CSVs, one firewall allowlist. Every quarter the merged file grows entries that three people added without talking to each other, and the person doing the merge (me, this time) ends up eyeballing CIDRs at midnight.
So I wrote allowlist_merge.py: a stdlib-only Python script that takes any number of team CSVs, dedupes them, flags ranges claimed by more than one owner, finds entries already swallowed by a wider subnet, and emits a clean nftables set. The whole thing runs in under a second and the exit code is the number of findings, so cron can alarm on a messy merge.
Source links
The two modules doing all the work: Python ipaddress docs and Python csv docs. If your firewall side is nftables, my earlier walkthrough is here.
The input files
Each team submits a CSV with network,owner,note columns. Three of my synthetic demo files look like this. All addresses are RFC 5737 documentation ranges, by the way, nothing here is a real allowlist.
network,owner,note
203.0.113.10,itops,jenkins CI
203.0.113.0/27,itops,office VPN pool
203.0.113.44,itops,vpn exit
198.51.100.7,itops,monitoring probe
192.0.2.10,itops,old bastion (keep)network,owner,note
203.0.113.0/25,office,HQ subnet
203.0.113.44,office,same vpn exit
203.0.113.130,office,reception wi-fiThe vendors file is where it gets fun: one entry duplicates the bastion, one is covered by their own CDN range, and somebody pasted a hostname into the IP column.
network,owner,note
192.0.2.0/28,vendors,partner B2B range
192.0.2.10,vendors,their api egress
203.0.113.200/29,vendors,saas scanner
198.51.100.0/24,vendors,cdn nodes
webhook.vendor.example,vendors,"oops, hostname"The script
One detail worth knowing up front: ipaddress.ip_network() with strict=False silently normalises a host address like 203.0.113.44 that a team pasted as a bare IP into whatever subnet it belongs to, which is exactly the bug class we are hunting, so the script never passes strict=False for single hosts and relies on the interface-style normalisation shown below to fold /32 forms consistently.
#!/usr/bin/env python3
"""Merge team IP allowlist CSVs, dedupe, and flag redundancy.
Each CSV needs columns: network, owner, note.
Exit code = number of findings (conflicts + redundant coverages).
"""
import argparse, csv, ipaddress, sys
from collections import defaultdict
from pathlib import Path
def normalise(raw):
net = ipaddress.ip_network(raw.strip(), strict=False)
if net.prefixlen == net.max_prefixlen: # /32 on IPv4 -> plain address
net = ipaddress.ip_network(f"{net.network_address}/32")
return net
def load(paths):
entries, bad = [], []
for path in paths:
with open(path, newline="") as fh:
for lineno, row in enumerate(csv.DictReader(fh), start=2):
try:
net = normalise(row["network"])
except ValueError:
bad.append((path.name, lineno, row["network"]))
continue
entries.append((net, row["owner"].strip(),
row["note"].strip(), Path(path).name, lineno))
return entries, bad
def main():
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("csvs", nargs="+", type=Path, help="allowlist CSV files")
ap.add_argument("--nft-file", type=Path, help="also write an nftables set")
args = ap.parse_args()
entries, bad = load(args.csvs)
# group identical networks
by_net = defaultdict(list)
for net, owner, note, fname, lineno in entries:
by_net[net].append((owner, note, fname, lineno))
nets = sorted(by_net, key=lambda n: (n.network_address.packed, n.prefixlen))
# exact duplicates across owners -> conflict report
conflicts = {n: rows for n, rows in by_net.items()
if len({r[0] for r in rows}) > 1}
# redundant coverage: entry fully inside a broader entry owned elsewhere
redundant = []
for n in nets:
for b in nets:
if b.version != n.version or b.prefixlen >= n.prefixlen:
continue
if n.subnet_of(b):
owners_b = sorted({r[0] for r in by_net[b]})
redundant.append((n, b, owners_b))
break
print(f"parsed {len(entries)} entries from {len(args.csvs)} files")
print(f"unique networks after dedupe: {len(nets)}")
if bad:
print(f"skipped {len(bad)} unparseable rows:")
for f, ln, val in bad:
print(f" {f}:{ln} {val!r} is not an IP or CIDR")
print(f"\nconflicts (same range claimed by >1 owner): {len(conflicts)}")
for n, rows in sorted(conflicts.items(), key=lambda kv: str(kv[0])):
who = sorted({(r[0], r[2], r[3]) for r in rows})
print(f" {n} claimed by " + ", ".join(f"{o} ({f}:{l})" for o, f, l in who))
print(f"\nredundant entries (already covered by a wider range): {len(redundant)}")
for n, b, owners in redundant:
print(f" {n} is inside {b} (owner: {','.join(sorted(owners))}) -> drop candidate")
keep = [n for n in nets if n not in {r[0] for r in redundant}]
print(f"\nfinal list: {len(keep)} entries\n")
for n in keep:
owners = "/".join(sorted({r[0] for r in by_net[n]}))
print(f" {str(n.network_address):>15} prefix /{n.prefixlen:<3} owner {owners}")
if args.nft_file:
lines = ["table inet fw {", " set allowlist {",
" type ipv4_addr", " flags interval", " elements = {"]
body = [f" {n}" for n in keep]
lines += [l + ("," if i < len(body) - 1 else "") for i, l in enumerate(body)]
lines += [" }", " }", "}", ""]
args.nft_file.write_text("\n".join(lines))
print(f"wrote nftables set to {args.nft_file}")
findings = len(conflicts) + len(redundant)
print(f"\n{findings} finding(s)")
sys.exit(min(findings, 250))
if __name__ == "__main__":
main()Running it
$ cd /etc/firewall/allowlists
$ python3 allowlist_merge.py data/*.csv --nft-file out.nft
echo $?On the demo data the report looks like this. This is the real captured stdout, not a mockup.
parsed 12 entries from 3 files
unique networks after dedupe: 10
skipped 1 unparseable rows:
vendors.csv:6 'webhook.vendor.example' is not an IP or CIDR
conflicts (same range claimed by >1 owner): 2
192.0.2.10/32 claimed by itops (itops.csv:6), vendors (vendors.csv:3)
203.0.113.44/32 claimed by itops (itops.csv:4), office (office.csv:3)
redundant entries (already covered by a wider range): 5
192.0.2.10/32 is inside 192.0.2.0/28 (owner: vendors) -> drop candidate
198.51.100.7/32 is inside 198.51.100.0/24 (owner: vendors) -> drop candidate
203.0.113.0/27 is inside 203.0.113.0/25 (owner: office) -> drop candidate
203.0.113.10/32 is inside 203.0.113.0/25 (owner: office) -> drop candidate
203.0.113.44/32 is inside 203.0.113.0/25 (owner: office) -> drop candidate
final list: 5 entries
192.0.2.0 prefix /28 owner vendors
198.51.100.0 prefix /24 owner vendors
203.0.113.0 prefix /25 owner office
203.0.113.130 prefix /32 owner office
203.0.113.200 prefix /29 owner vendors
wrote nftables set to out.nft
7 finding(s)
Twelve entries in, ten unique networks, and seven findings: two conflicts where two teams both claim the same address, plus five drop candidates that a wider range already covers. The 192.0.2.10 bastion that IT swore was special? The vendor route already lets them in, and the note column tells you who to argue with.
The nftables output
The generated set is a single --nft-file away, and you can validate it before loading. Same five kept entries as the report:
table inet fw {
set allowlist {
type ipv4_addr
flags interval
elements = {
192.0.2.0/28,
198.51.100.0/24,
203.0.113.0/25,
203.0.113.130/32,
203.0.113.200/29
}
}
}$ nft --check -f out.nft && echo ok
ok
One bug I hit while verifying
My first version wrote a broken nft file. The elements = { block never got its closing brace because I reused the set and table closers as one list, so the output was two braces deep for a three-level structure. nft --check caught it immediately, which is why that validation line is in the post. If you write this kind of generator, always pipe the output through the tool that consumes it before you trust the report.
Making it part of the routine
The exit code equals the finding count, so a nightly cron entry that mails you only on a non-zero result does the alerting for free. That pairs nicely with the cron and systemd timer hygiene audit I wrote about, and if your allowlists are actually SSH-based, the sshd_config hardening audit catches the other half of this problem.
Run it before your next firewall change. The twenty seconds it spends finding duplicates beats the twenty minutes of Slack archaeology.