Nginx Access Log Analysis with Bash and Python

– min read

Hi, last month I was reviewing a client staging box that the owner believed was “under a DDoS”. It was not a DDoS. It was one scanner with a grudge and six other bots doing their morning rounds, and the whole story was sitting in a plain text file nobody had opened: /var/log/nginx/access.log. You do not need a SIEM to find out who is hammering your server. You need awk, one small Python script, and ten minutes.

Sources: nginx | GNU awk manual | Python collections docs

The default nginx welcome page on a fresh install
The default nginx page. Source: Wikimedia Commons (CC BY-SA 4.0).


Know your log format first

The combined format, the nginx default, puts the same fields in the same columns on almost every server you will ever touch:

Fieldawk positionMeaning
Remote addr$1Client IP, your primary suspect column
Timestamp$4Looks like [02/Nov/2026:13:10:01, note the bracket
Request$6 $7 $8Method, path, protocol
Status$9200, 404, that 500 you forgot about
Bytes$10Response size, useful for finding scrapers

One honest warning: every command below assumes combined format and no quotes inside the URI. If your log_format is custom, check a couple of lines before trusting the column numbers.

HTTP response headers from nginx fetched with curl on Ubuntu
curl reading headers from an nginx site. Source: Wikimedia Commons (CC BY-SA 3.0).


The five one-liners I run first

This is my whole triage. Who is talking the most:

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
   3036 203.0.113.1
   3033 203.0.113.6
   3014 203.0.113.9
   3009 203.0.113.4
   3001 203.0.113.2
   2998 203.0.113.10
   2949 203.0.113.11
   2930 203.0.113.8
   2919 203.0.113.3
   2915 203.0.113.7

Top talkers alone are boring, real users cluster at the top. The interesting question is what got rejected and where. This finds the paths that drew 404s and 403s, which is the bot equivalent of a fingerprint:

awk '$9 == "404" || $9 == "403" {print $7}' access.log \
  | sort | uniq -c | sort -rn | head -6
   1086 /.env
   1071 /wp-login.php
   1060 /vendor/phpunit/
   1056 /cgi-bin/test.cgi
   1051 /phpmyadmin/
   1040 /xmlrpc.php

That list is not traffic, it is a vulnerability scan wearing a browser user agent. Next, when did they arrive? A flat 24/7 rate is a bot, a morning ramp is humans:

awk -F'[:[]' '{print $3}' access.log | sort | uniq -c | head -6
   1895 00
   1869 01
   1822 02
   1850 03
   1815 04
   1870 05

Just over 1,800 requests every hour, at 3 in the morning, with zero variance. A human does not browse your site that evenly, a cron job does.

Which IP family is probing? Narrow the filter, then count requests per day:

awk '$1 ~ /^45\.148\.10\./ {print substr($4,2,11)}' access.log \
  | sort | uniq -c
   1448 01/Nov/2026
   1422 02/Nov/2026
   1464 03/Nov/2026
   1340 04/Nov/2026
   1424 05/Nov/2026
   1447 06/Nov/2026
    771 07/Nov/2026

About 1,400 hits a day from a single /24. And when your logs rotate, the same one-liners work with zero changes if you stream them:

zcat access.log.*.gz | awk '{print $1}' | sort | uniq -c | sort -rn | head


The 30 line Python script

awk is perfect for questions you have already answered once. Python is better for the ones you keep asking. This is the script I drop on every new server now: no dependencies, standard library only, it answers the three triage questions in a single pass:

#!/usr/bin/env python3
"""Summarise an nginx access log: hits per IP, per status, and the
paths that drew the most 404s. Usage: ./topoffenders.py access.log"""
import sys
from collections import Counter
from http import HTTPStatus

hits, status, dead = Counter(), Counter(), Counter()
with open(sys.argv[1]) as fh:
    for line in fh:
        parts = line.split('"')
        if len(parts) < 3:
            continue
        meta = parts[2].split()
        ip = line.split(None, 1)[0]
        hits[ip] += 1
        if len(meta) >= 1:
            status[meta[0]] += 1
        if len(meta) >= 1 and meta[0] == "404":
            dead[parts[1].split()[1]] += 1

print(f"total lines parsed: {sum(hits.values())}")
print(f"unique IPs: {len(hits)}")
print("\ntop 10 talkers")
for ip, n in hits.most_common(10):
    print(f"{n:>7}  {ip}")
print("\nstatus mix")
for code, n in status.most_common():
    phrase = HTTPStatus(int(code)).phrase if code.isdigit() else "?"
    print(f"{n:>7}  {code} {phrase}")
print("\ntop 404 magnets")
for path, n in dead.most_common(5):
    print(f"{n:>7}  {path}")

Run it against the log and you get the whole health check instantly:

$ python3 topoffenders.py access.log
total lines parsed: 42000
unique IPs: 15

top 10 talkers
   3036  203.0.113.1
   3033  203.0.113.6
   3014  203.0.113.9

status mix
  27596  200 OK
   6567  404 Not Found
   3020  500 Internal Server Error
   2989  301 Moved Permanently
   1828  403 Forbidden

top 404 magnets
    852  /.env
    848  /phpmyadmin/
    845  /wp-login.php

A debugging note if you extend this: HTTPStatus wants an int, not the string you pulled out of the log, so HTTPStatus(code) throws ValueError: 403 is not a valid HTTPStatus even though 403 is obviously a valid status. It is the type, not the number. I hit this exact wall writing the script and int(code) fixed it.


From findings to action

Once an IP earns its place in your 404 magnets list, the quick kill is a firewall rule. Review the list first, then:

awk '$1 ~ /^45\.148\.10\./ {print $1}' access.log \
  | sort -u | xargs -n1 sudo ufw deny
sudo ufw status numbered

For something that keeps itself current, fail2ban with the nginx-botsearch jail does this job on a timer and un-bans later, which is usually what you actually want.


Gotchas

TrapReality check
Wrong column numbersCheck log_format in /etc/nginx/nginx.conf before trusting $9
Quotes in URIsA path containing a quote character shifts the fields, the Python version handles it via the quote-split trick
CDN trafficBehind Cloudflare or similar, $1 is the CDN, log the real client IP from a header instead
IPv6 visitorsThey contain colons, the hour extraction with -F'[:[' still works, plain cut -d: does not


People ask why I still grep logs when observability stacks exist. Because a log file is the only monitoring source that cannot lie about its own ingestion lag. Thirty seconds with awk on a box I have never seen before tells me more than an hour of clicking someone else’s dashboard. And one honest footnote: the demo log here is synthetic, 42,000 lines generated with realistic scanner behavior, because a production access log is client data and client data stays out of blog posts. The commands and every output are the same ones I run on real logs, only the IPs in them are from TEST-NET.

Mohamad ZulfahmyCybersecurity · OffSec · Entrepreneur
About me ↗
ON THIS PAGE
On this page
Xin

Keep reading