This article provides configurations and shell scripts to run nftables – the newer/better version to replace iptables. This covers rate limiting and blocking. Code for Debian 13 included.

NFTABLES & Security

Note: If you want to just get things working – this should do it, and you can go back to the .txt data files to reset values.

We use /home/ec2-user/nft (or admin in Debian) for the scripts, and /var/log/nft for configuration and log files.

One of these is /var/log/nft/no_files.txt where our script searches nginx/apache2 access.log files for bot attempts to find files that do not exist – or even if they do we want to block them for having tried. You have to edit this for your own blocking.

An example could be that yu see access.log shown an attempt for “.git”. You could go to /var/www/ (or wherever) and use the Linux/Unix find command to search for the string .git. If it does not exist, you know this is a name you can use to block.

This scripting chooses to block most bad actors for 72 hours. If we want permanent IP address or IP CIDR ranges blocked, these go in the file /var/log/nft/block.txt. There is clearly need for improvement to the first cut of these scripts, but we have a basic configuration that works.

We also prevent bots that do things like issuing hundreds of commands within a second.

Keep in mind how fatal it is to your server to open the SSH port 22 to anyone. This will be flooded and freeze your resources, so only open the port to your IP address.

Any configurations for rate limiting and blocking need testing against the apps you run. For instance, you should only block the use of wp-login.php if you have an alternative login activated by a WordPress plugin.

You must add your server’s public IP address and your own to the whitelist.

Our Design

Whitelist your own nominated IP addresses such as your broadband provider, a few search engines like Google, and update google allowed IP’s regularly via the google shell script.

Block Countries – (IP Addresses available via IP2LOCATION’s lite website) – Namely Russia, China, India (as examples). You can modify the scripts for other countries. Country listings can be huge and messy. We run the script with a tidy-up on these, and we run with multiple chunks of IP addresses for the whole process to actually work.

Block nominated bad domains

Block a list of known bad actors (Permanent blocks)

Time limit blocking of bad actors where tests are done on /var/log/nginx/access.log for folders that do not exist, but with minimum repeats allowed. e.g. why block someone if they repeat an attempt to access a nominated file by mistake.

Do incremental checks throughout the day via crontab.

Perform safe rate limiting on ports.

/var/log/archive keeps some previous error/access logs in case a review of those help.

chatGPT Documentation of this Solution

We use the new scripts /home/ec2-user/firewall-nft.sh and firewall-incremental.sh. We make use of /var/log/nft files as well, and /tmp.
Here is an inventory of the solution:

Below is a structured inventory and functional breakdown of both scripts and how they work together.

You effectively have:

firewall-nft.sh → full rebuild (weekly, authoritative)
firewall-incremental.sh → fast reactive updates (every 10 minutes)

Both operate on an nftables table:
table inet filter

1️⃣ High-Level Architecture

Weekly (Full Rebuild)

firewall-nft.sh

Deletes and recreates the entire inet filter table
Rebuilds all sets from scratch
Re-parses logs
Re-loads static and dynamic blocklists
Re-resolves bad domains
Reapplies rate limiting rules

This ensures drift, stale entries, and nft timeouts are cleaned up weekly.

Every 10 Minutes (Incremental)

firewall-incremental.sh

Does not recreate the table
Only adds new IPs discovered in logs
Updates:
- no_file_block
- optional_block
Persists discovered IPs into flat files for the weekly rebuild

This script is lightweight and reactive.

2️⃣ nftables Structure Being Built

Both scripts operate on:

table inet filter

Sets Created (weekly script)

Set Name	Purpose	Timeout
whitelist	Allowed IPs	none
country_ru	Russia IP ranges	none
country_cn	China IP ranges	none
country_in	India IP ranges	none
manual_block	Admin-added IPs	72h
optional_block	Soft auto-block	72h
domain_block	Resolved bad domains	none
no_file_block	Aggressive scanner block	72h

Input Chain Behavior

Policy: drop

Order matters:

Allow loopback
Allow whitelist
Allow established
Drop invalid
Drop from all block sets (manual, optional, nofile, domain, RU, CN, IN)
Allow:
- 80 (HTTP)
- 443 (HTTPS via https_protect chain)
- 22 (SSH)

HTTPS Protection Chain

ct count over 300 drop
limit rate over 30/second drop
accept

This:

Drops IPs with more than 300 concurrent connections
Drops if exceeding 30 new connections/sec
Otherwise accepts

Basic anti-flood protection.

3️⃣ Complete File Inventory

All files used under /var/log/nft

🌍 Country Lists

File	Used By	Purpose
`/var/log/nft/russia.txt`	weekly	RU IPv4 ranges
`/var/log/nft/china.txt`	weekly	CN IPv4 ranges
`/var/log/nft/india.txt`	weekly	IN IPv4 ranges

Loaded into respective country sets.

🛑 Manual / Admin Control

File	Used By	Purpose
`/var/log/nft/block.txt`	weekly	Manual IP blocks
`/var/log/nft/optional-block.txt`	weekly	Predefined optional blocks

These populate:

manual_block
optional_block

Manual and optional sets have 72h timeout.

🔎 Log-Based Detection

File	Used By	Purpose
`/var/log/nginx/access.log`	both	Source of detected abusive IPs
`/var/log/nft/no_file.txt`	both	Patterns for mandatory blocking
`/var/log/nft/opt_patterns.txt`	incremental	Patterns for optional blocking

How They Work

Mandatory (no_file_block)

If a pattern in no_file.txt appears:

Weekly script: blocks IP if ≥50 hits
Incremental script: blocks immediately

IPs added to:

nft set no_file_block
/var/log/nft/mandatory-ipv4.txt

Optional (optional_block)

If a pattern in opt_patterns.txt appears:

Only if count ≥ 3 (OPTIONAL_NSTRIKE)
Added to:
- nft set optional_block
- /var/log/nft/optional-ipv4.txt

💾 Persisted Dynamic IP Files

File	Used By	Purpose
`/var/log/nft/mandatory-ipv4.txt`	both	Persist auto-mandatory IPs
`/var/log/nft/optional-ipv4.txt`	both	Persist optional IPs

These ensure weekly rebuild restores incremental discoveries.

🌐 Domain Blocking

File	Used By	Purpose
`/var/log/nft/bad-domains.txt`	weekly	Domains to resolve and block

Weekly script:

Runs dig +short
Extracts IPv4
Adds to domain_block

Note: these are not periodically re-resolved except weekly.

🔎 Whitelist

Source	Used By
Hardcoded IP list	weekly
`/var/log/nft/search-engine-whitelist.txt`	weekly

Whitelist overrides all blocking logic.

Incremental script checks whitelist before blocking.

🔐 Locking

File	Used By	Purpose
`/var/log/nft/firewall-nft-incremental.lock`	incremental	Prevents concurrent runs

Uses flock.

4️⃣ What Each Script Is Doing Internally

🔵 firewall-nft.sh (Weekly Full Reset)

Step-by-step

Delete existing inet filter table
Generate fresh nft ruleset
Load base structure
Populate:
- whitelist
- country lists
- manual blocks
- optional blocks
- persisted dynamic IPs
Resolve bad domains
Parse logs for heavy no-file attackers (≥50 hits)
Load them into no_file_block
Remove temp directory

It rebuilds the entire firewall state from disk + logs.

🟢 firewall-incremental.sh (Every 10 Minutes)

Step-by-step

Exit if table missing
Lock execution
Parse access.log
For each pattern:
- Extract IPs
- Skip whitelist
- Skip already blocked
- Add to nft set
- Append to persistent file
Unlock

It never deletes anything.
Only appends.

5️⃣ Behavioral Design Pattern

You built a hybrid model:

Authoritative weekly rebuild
Fast reactive incremental layer
Timeout-based decay (72h)
Persistent flat file recovery
Country-level blocking
Domain-to-IP resolution
Log-driven auto-ban

This is essentially a lightweight fail2ban-style firewall built directly on nftables.

6️⃣ Important Observations

1️⃣ Timeout Behavior

Sets with timeout:

manual_block
optional_block
no_file_block

These expire automatically after 72h.

But since you also persist them to flat files:

Weekly rebuild re-adds them
Which resets timeout

So effective timeout becomes:

72h unless weekly rebuild reloads it sooner

2️⃣ Log Parsing Behavior

Weekly script:

no_file_block requires ≥50 hits

Incremental script:

Mandatory patterns block immediately
Optional patterns require ≥3 hits

So incremental is more aggressive.

3️⃣ Possible Edge Conditions

Domain IPs can change between weekly runs
Access.log growth may slow grep parsing over time
No log rotation offset handling
No IPv6 support
Country lists must be manually maintained

7️⃣ Summary Diagram (Conceptual Flow)

NGINX access.log
       │
       ├── firewall-incremental (10 min)
       │       ├── add to nft
       │       └── persist to file
       │
       └── firewall-nft (weekly rebuild)
               ├── rebuild nft table
               ├── reload flat files
               ├── reload country lists
               ├── resolve domains
               └── re-parse logs

8️⃣ Final Inventory Checklist

You are using:

nft

/usr/sbin/nft

System tools

grep
awk
sort
uniq
dig
flock
mktemp

Logs

/var/log/nginx/access.log

Data directory

/var/log/nft/

Containing:

russia.txt
china.txt
india.txt
block.txt
optional-block.txt
bad-domains.txt
no_file.txt
opt_patterns.txt
search-engine-whitelist.txt
mandatory-ipv4.txt
optional-ipv4.txt
firewall-nft-incremental.lock

Install nftables

dnf install nftables
dnf install nftables-devel
dnf install ipcalc

We should end up with:
nftables.aarch64                            
nftables-devel.aarch64            
python3-nftables.aarch64   

iptables-nft.aarch64 may be hanging around but we do not use iptables - if you used to use it make sure it is not running,

You should have dnsutils and that dig is available as a command.

For DEBIAN:

sudo apt update
sudo apt install ipcalc -y
apt install dnsutils -y

The Scripts (linux2023)

See the toggle section /var/log/nft below for the files you need to place in /var/log/nft, with examples of russia and china ip addresses, etc.

firewall-nft.sh

cd /home/ec2-user/nft

vi firewall-nft.sh

#!/bin/sh
set -e

MODE="${1:-live}"

if [ "$MODE" = "dry-run" ]; then
    DRYRUN=1
else
    DRYRUN=0
fi

nft_cmd() {
    if [ "$DRYRUN" -eq 1 ]; then
        echo "nft $*"
    else
        "$NFT" "$@"
    fi
}

### -------------------------------
### CONFIGURATION
### -------------------------------

TABLE="inet filter"
NFT="/usr/sbin/nft"

SET_WHITELIST="whitelist"
SET_RU="country_ru"
SET_CN="country_cn"
SET_IN="country_in"
SET_MANUAL="manual_block"
SET_OPTIONAL="optional_block"
SET_DOMAIN="domain_block"
SET_NOFILE="no_file_block"

RU_FILE="/var/log/nft/russia.txt"
CN_FILE="/var/log/nft/china.txt"
IN_FILE="/var/log/nft/india.txt"
MANUAL_FILE="/var/log/nft/block.txt"
OPTIONAL_FILE="/var/log/nft/optional-block.txt"
BAD_DOMAINS_FILE="/var/log/nft/bad-domains.txt"
NO_FILE_PATTERNS="/var/log/nft/no_file.txt"
SEARCH_ENGINE_WHITELIST="/var/log/nft/search-engine-whitelist.txt"

MANDATORY_IP_FILE="/var/log/nft/mandatory-ipv4.txt"
OPTIONAL_IP_FILE="/var/log/nft/optional-ipv4.txt"

ACCESS_LOG="/var/log/nginx/access.log"
MANUAL_TIMEOUT="72h"

WHITELIST="
127.0.0.1
117.20.68.0/22
3.107.133.61
172.31.35.212
"

TMPDIR="$(mktemp -d)"
NFT_FILE="$TMPDIR/nftables.conf"

### -------------------------------
### REMOVE EXISTING TABLE
### -------------------------------

nft_cmd delete table inet filter 2>/dev/null || true

### -------------------------------
### GENERATE RULESET
### -------------------------------

cat > "$NFT_FILE" <<EOF
#!/usr/sbin/nft -f

table inet filter {

    set $SET_WHITELIST {
        type ipv4_addr
        flags interval
    }

    set $SET_RU {
        type ipv4_addr
        flags interval
    }

    set $SET_CN {
        type ipv4_addr
        flags interval
    }

    set $SET_IN {
        type ipv4_addr
        flags interval
    }

    set $SET_MANUAL {
        type ipv4_addr
        flags interval
        timeout $MANUAL_TIMEOUT
    }

    set $SET_OPTIONAL {
        type ipv4_addr
        flags interval
        timeout $MANUAL_TIMEOUT
    }

    set $SET_DOMAIN {
        type ipv4_addr
        flags interval
    }

    set $SET_NOFILE {
        type ipv4_addr
        flags interval
        timeout $MANUAL_TIMEOUT
    }

    chain input {
        type filter hook input priority 0;
        policy drop;

        iif lo accept
        ip saddr @$SET_WHITELIST accept

        ct state established,related accept
        ct state invalid drop

        ip saddr @$SET_MANUAL drop
        ip saddr @$SET_OPTIONAL drop
        ip saddr @$SET_NOFILE drop
        ip saddr @$SET_DOMAIN drop
        ip saddr @$SET_RU drop
        ip saddr @$SET_CN drop
        ip saddr @$SET_IN drop

        tcp dport 80 ct state new accept
        tcp dport 443 tcp flags syn ct state new jump https_protect
        tcp dport 22 ct state new accept
    }

    chain https_protect {
        ct count over 300 drop
        limit rate over 30/second drop
        accept
    }
}
EOF

### -------------------------------
### LOAD RULESET
### -------------------------------

if [ "$DRYRUN" -eq 1 ]; then
    "$NFT" -c -f "$NFT_FILE"
else
    "$NFT" -f "$NFT_FILE"
fi

### -------------------------------
### FUNCTIONS
### -------------------------------

load_set() {
    file="$1"
    set="$2"
    CHUNK_SIZE=500
    [ ! -s "$file" ] && return

    TMP_SORTED=$(mktemp)
    sort -Vu "$file" > "$TMP_SORTED"

    buffer=""
    count=0
    while read -r ip; do
        case "$ip" in ""|\#*) continue ;; esac
        buffer="${buffer:+$buffer, }$ip"
        count=$((count + 1))
        if [ "$count" -ge "$CHUNK_SIZE" ]; then
            nft_cmd add element "$TABLE" "$set" { $buffer } 2>/dev/null || true
            buffer=""
            count=0
        fi
    done < "$TMP_SORTED"

    [ -n "$buffer" ] && nft_cmd add element "$TABLE" "$set" { $buffer } 2>/dev/null || true
    rm -f "$TMP_SORTED"
}

load_domains() {
    [ ! -s "$BAD_DOMAINS_FILE" ] && return
    while read -r d; do
        case "$d" in ""|\#*) continue ;; esac
        dig +short "$d" | grep -E '^[0-9]' | while read -r ip; do
            nft_cmd add element "$TABLE" "$SET_DOMAIN" { $ip } 2>/dev/null || true
        done
    done < "$BAD_DOMAINS_FILE"
}

load_no_file_attempts() {
    logfile="$1"
    pattern_file="$2"
    set="$3"
    [ ! -s "$pattern_file" ] || [ ! -s "$logfile" ] && return

    while read -r pattern; do
        case "$pattern" in ""|\#*) continue ;; esac
        grep -F "$pattern" "$logfile" | \
        grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | \
        sort | uniq -c | awk '$1 >= 50 {print $2}' | \
        while read -r ip; do
            nft_cmd get element "$TABLE" "$SET_WHITELIST" { $ip } >/dev/null 2>&1 && continue
            nft_cmd add element "$TABLE" "$set" { $ip } 2>/dev/null || true
        done
    done < "$pattern_file"
}

### -------------------------------
### LOAD SETS
### -------------------------------

for ip in $WHITELIST; do
    nft_cmd add element "$TABLE" "$SET_WHITELIST" { $ip } 2>/dev/null || true
done

load_set "$SEARCH_ENGINE_WHITELIST" "$SET_WHITELIST"
load_set "$RU_FILE" "$SET_RU"
load_set "$CN_FILE" "$SET_CN"
load_set "$IN_FILE" "$SET_IN"
load_set "$MANUAL_FILE" "$SET_MANUAL"
load_set "$OPTIONAL_FILE" "$SET_OPTIONAL"
load_set "$MANDATORY_IP_FILE" "$SET_NOFILE"
load_set "$OPTIONAL_IP_FILE" "$SET_OPTIONAL"

load_domains
load_no_file_attempts "$ACCESS_LOG" "$NO_FILE_PATTERNS" "$SET_NOFILE"

rm -rf "$TMPDIR"

[save and exit]
(chmod 777 and root:user)

firewall-incremental.sh:

cd /home/ec2-user/nft

vi firewall-incremental.sh

#!/bin/sh
set -e

MODE="${1:-live}"

if [ "$MODE" = "dry-run" ]; then
    DRYRUN=1
else
    DRYRUN=0
fi

nft_cmd() {
    if [ "$DRYRUN" -eq 1 ]; then
        echo "nft $*"
    else
        "$NFT" "$@"
    fi
}

TABLE="inet filter"

SET_NOFILE="no_file_block"
SET_MANUAL="manual_block"
SET_OPTIONAL="optional_block"
SET_WHITELIST="whitelist"

NFT="/usr/sbin/nft"
ACCESS_LOG="/var/log/nginx/access.log"
NO_FILE_PATTERNS="/var/log/nft/no_file.txt"
OPT_PATTERNS="/var/log/nft/opt_patterns.txt"

MANDATORY_IP_FILE="/var/log/nft/mandatory-ipv4.txt"
OPTIONAL_IP_FILE="/var/log/nft/optional-ipv4.txt"

LOCKFILE="/var/log/nft/firewall-nft-incremental.lock"

OPTIONAL_NSTRIKE=3

nft_cmd list table inet filter >/dev/null 2>&1 || exit 0

touch "$MANDATORY_IP_FILE" "$OPTIONAL_IP_FILE"
chmod 600 "$MANDATORY_IP_FILE" "$OPTIONAL_IP_FILE"

exec 9>"$LOCKFILE"
flock -n 9 || exit 0

[ ! -s "$ACCESS_LOG" ] && exit 0

### Mandatory blocks
while read -r pattern; do
    case "$pattern" in ""|\#*) continue ;; esac
    grep -F "$pattern" "$ACCESS_LOG" | \
    grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -Vu | \
    while read -r ip; do
        nft_cmd get element "$TABLE" "$SET_WHITELIST" { $ip } >/dev/null 2>&1 && continue
        nft_cmd get element "$TABLE" "$SET_NOFILE" { $ip } >/dev/null 2>&1 && continue

        nft_cmd add element "$TABLE" "$SET_NOFILE" { $ip } 2>/dev/null || continue
        grep -qx "$ip" "$MANDATORY_IP_FILE" || echo "$ip" >> "$MANDATORY_IP_FILE"
    done
done < "$NO_FILE_PATTERNS"

### Optional blocks
[ -s "$OPT_PATTERNS" ] && \
while read -r pattern; do
    case "$pattern" in ""|\#*) continue ;; esac
    grep -F "$pattern" "$ACCESS_LOG" | \
    grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq -c | \
    awk -v n="$OPTIONAL_NSTRIKE" '$1>=n {print $2}' | \
    while read -r ip; do
        nft_cmd get element "$TABLE" "$SET_WHITELIST" { $ip } >/dev/null 2>&1 && continue
        nft_cmd get element "$TABLE" "$SET_OPTIONAL" { $ip } >/dev/null 2>&1 && continue

        nft_cmd add element "$TABLE" "$SET_OPTIONAL" { $ip } 2>/dev/null || continue
        grep -qx "$ip" "$OPTIONAL_IP_FILE" || echo "$ip" >> "$OPTIONAL_IP_FILE"
    done
done < "$OPT_PATTERNS"

flock -u 9

[save and exit + permissions]

google-whitelist.sh

#!/bin/sh
set -e

### -------------------------------
### Configuration
### -------------------------------
SEARCH_ENGINE_WHITELIST="/var/log/nft/search-engine-whitelist.txt"
LOG="/var/log/nft/update-googlebot-whitelist.log"

echo "$(date -Is) | Updating Googlebot IPv4 addresses..." | tee -a "$LOG"

TMPFILE=$(mktemp)
TMPFILE_GOOGLE=$(mktemp)

### -------------------------------
### Fetch current Googlebot IPv4 only
### -------------------------------
GOOGLEBOT_JSON="https://www.gstatic.com/ipranges/goog.json"

curl -s "$GOOGLEBOT_JSON" | \
    jq -r '.prefixes[] | select(.service=="Googlebot" and has("ipv4Prefix")) | .ipv4Prefix' | \
    grep -v null > "$TMPFILE_GOOGLE"

### -------------------------------
### Keep non-Googlebot entries from existing whitelist
### -------------------------------
if [ -s "$SEARCH_ENGINE_WHITELIST" ]; then
    grep -v -f "$TMPFILE_GOOGLE" "$SEARCH_ENGINE_WHITELIST" >> "$TMPFILE"
fi

### -------------------------------
### Add latest Googlebot entries
### -------------------------------
cat "$TMPFILE_GOOGLE" >> "$TMPFILE"

### -------------------------------
### Deduplicate and save
### -------------------------------
sort -u "$TMPFILE" > "$SEARCH_ENGINE_WHITELIST"
rm -f "$TMPFILE" "$TMPFILE_GOOGLE"

COUNT=$(wc -l < "$SEARCH_ENGINE_WHITELIST")
echo "$(date -Is) | Googlebot IPv4 whitelist updated. Total entries now: $COUNT" | tee -a "$LOG"

### -------------------------------
### Optional: Reload firewall-nft.sh
### -------------------------------
# /home/ec2-user/nft/firewall-nft.sh

flush-ruleset.sh (useful in testing to remove all rules)

#!/bin/sh
nft flush ruleset
exit

list.sh (useful to list all the rules)

#!/bin/sh
 nft list ruleset
exit

Clean Log FIles etc.

We use this for general log cleanups, but also in conjunction with nft.

cleanup-logs.sh

cd /home/ec2-user

vi cleanup-logs.sh

#!/bin/bash
# cleanup-logs.sh
# Safe log archival + truncation with locking and retention
# Archives logs BEFORE truncation
# Retains archives ~30 days
# No disruptive service restarts

set -euo pipefail

trap 'echo "ERROR at line $LINENO: $BASH_COMMAND" >&2' ERR

LOCKFILE="/var/run/cleanup-logs.lock"
ARCHIVE_DIR="/var/log/archive"
DATE="$(date +%Y-%m-%d)"
RETENTION_DAYS=30

exec 200>"$LOCKFILE"
flock -n 200 || exit 0

mkdir -p "$ARCHIVE_DIR"

# -------------------------
# Helper functions
# -------------------------

truncate_if_exists() {
    local file="$1"
    if [[ -f "$file" ]]; then
        /usr/bin/truncate -s 0 "$file" || true
    fi
}

archive_and_truncate() {
    local src="$1"
    local name="$2"

    if [[ -f "$src" ]]; then
        cp -p "$src" "$ARCHIVE_DIR/${name}-${DATE}.log" || true
        /usr/bin/truncate -s 0 "$src" || true
    fi
}

# =========================================================
# ---- SYSTEM LOG CLEANUP (no archiving needed) ----
# =========================================================

# ---- Cloud-init Logs ----
truncate_if_exists "/var/log/cloud-init.log"
truncate_if_exists "/var/log/cloud-init-output.log"

# ---- DNF Logs ----
truncate_if_exists "/var/log/dnf.log"
truncate_if_exists "/var/log/dnf.log.1"
truncate_if_exists "/var/log/dnf.rpm.log"
truncate_if_exists "/var/log/dnf.librepo.log"
truncate_if_exists "/var/log/dnf.librepo.log.1"

# ---- Hawkey logs (glob-safe) ----
shopt -s nullglob
for f in /var/log/hawkey*; do
    truncate_if_exists "$f"
done
shopt -u nullglob

# ---- Lastlog / wtmp ----
truncate_if_exists "/var/log/lastlog"
truncate_if_exists "/var/log/wtmp"

# ---- Optional: sysstat (sar) ----
if [[ -d "/var/log/sa" ]]; then
    rm -f /var/log/sa/* >/dev/null 2>&1 || true
fi

# ---- SSSD LOGS ----
# if [[ -d /var/log/sssd ]]; then
    # for f in /var/log/sssd/*.log; do
        # [[ -f "$f" ]] && truncate -s 0 "$f" || true
    # done
# fi

# ---- journal logs ----
journalctl --flush --rotate --vacuum-time=1s >/dev/null 2>&1 || true

# Clean journal files without hardcoding machine-id
# You may need old logs, so clean manually at /var/log/journal/.../system@ and user-*
# These do take up a lot of disk space


# -------------------------
# ---- NGINX LOGS ----
# -------------------------
NGINX_DIR="/var/log/nginx"

if [[ -d "$NGINX_DIR" ]]; then
    archive_and_truncate "$NGINX_DIR/access.log"   "access"
    archive_and_truncate "$NGINX_DIR/error.log"    "error"
    archive_and_truncate "$NGINX_DIR/access.log.1" "access.1"
    archive_and_truncate "$NGINX_DIR/error.log.1"  "error.1"

    systemctl reload nginx >/dev/null 2>&1 || true

    rm -f "$NGINX_DIR"/*.gz >/dev/null 2>&1 || true
fi

# -------------------------
# -- /var/log/*.gz files --
# -------------------------
rm -f /var/log/*.gz >/dev/null 2>&1 || true

# -------------------------
# ---- MAIL LOGS ----
# -------------------------
archive_and_truncate "/var/log/mail.log" "mail"

# -------------------------
# ---- MariaDB LOGS ----
# -------------------------
archive_and_truncate "/var/log/mariadb/mariadb.log" "mariadb"

# -------------------------
# ---- PHP-FPM LOGS ----
# -------------------------
archive_and_truncate "/var/log/php-fpm/error.log" "php-fpm-error"

# -------------------------
# ---- Archive retention ----
# -------------------------
find "$ARCHIVE_DIR" -type f -name "*.log" -mtime +"$RETENTION_DAYS" -delete || true

# ---- Finished ----
echo "Log cleanup completed at $(date)"

exit 0

[save and exit + permissions]

crontab entries:

@reboot sleep 30 && sudo /home/ec2-user/nft/firewall-nft.sh >/dev/null 2>&1

# 1:30am each Sunday - certain rotation files in /var/log/archive
30 1 * * 0 sudo /home/ec2-user/cleanup-logs.sh > /dev/null 2>&1

# Every 10 minutes use nftables to update automatically
*/10 * * * * /home/ec2-user/nft/firewall-incremental.sh >/dev/null 2>&1

# weekly rebuild
0 2 * * 0 /home/ec2-user/nft/firewall-nft.sh >/dev/null 2>&1

# check google whitelist is current
0 4 1 * * /home/ec2-user/nft/google_whitelist.sh >/dev/null 2>&1

/var/log/nft files

These are the files we use in /var/log/nft (the scripts will create files initially as needed)
We don;t need to understand all of it, but you can edit data files to removes lists of IP addresses. This is to get us going. Install these in /var/log/nft:
drwxrwxrwx 4 root root 16384 Jan 28 13:25 nft
and all files in ./nft as:
-rwxrwxrwx 1 root ec2-user

2026-02-21      0.00 MB   file: bad-domains.txt
2026-02-21      0.01 MB   file: block.txt
2026-02-21      0.11 MB   file: china.txt
2026-02-21      0.00 MB  file: cidr.py
2026-02-21      0.24 MB   file: india.txt
2026-02-21      0.03 MB   file: iran.txt
2026-02-21      0.01 MB   file: mandatory-ipv4.txt
2026-02-21      0.00 MB   file: manual-blocked.txt
2026-02-21      0.00 MB   file: no_file.txt
2026-02-21      0.00 MB   file: northkorea.txt
2026-02-21      0.00 MB   file: opt_patterns.txt
2026-02-21      0.25 MB   file: russia.txt
2026-02-21      0.00 MB   file: search-engine-whitelist.txt

cidr.py is a useful(?) script that converts a huge list of ip addresses into a proper list, removing cross cidr numbering and duplicates. May be if use?

You can edit the firewall scripts to include other countries etc.

The Scripts (Debian 13)

firewall-nft.sh

#!/bin/sh
set -e

### -------------------------------
### CONFIGURATION
### -------------------------------

TABLE="inet filter"

# Sets
SET_WHITELIST="whitelist"
SET_RU="country_ru"
SET_CN="country_cn"
SET_IN="country_in"
SET_MANUAL="manual_block"
SET_OPTIONAL="optional_block"
SET_DOMAIN="domain_block"
SET_NOFILE="no_file_block"

# Static input files
RU_FILE="/var/log/nft/russia.txt"
CN_FILE="/var/log/nft/china.txt"
IN_FILE="/var/log/nft/india.txt"
MANUAL_FILE="/var/log/nft/block.txt"
OPTIONAL_FILE="/var/log/nft/optional-block.txt"
BAD_DOMAINS_FILE="/var/log/nft/bad-domains.txt"
NO_FILE_PATTERNS="/var/log/nft/no_file.txt"
SEARCH_ENGINE_WHITELIST="/var/log/nft/search-engine-whitelist.txt"

# Persistent dynamic IPs
MANDATORY_IP_FILE="/var/log/nft/mandatory-ipv4.txt"
OPTIONAL_IP_FILE="/var/log/nft/optional-ipv4.txt"

ACCESS_LOG="/var/log/nginx/access.log"
MANUAL_TIMEOUT="72h"

WHITELIST="
127.0.0.1
117.20.68.0/22
13.237.107.163
172.31.35.212
"

TMPDIR="$(mktemp -d)"
NFT_FILE="$TMPDIR/nftables.conf"
LOG="/var/log/nft/firewall-nft.log"
: > "$LOG"

### -------------------------------
### REMOVE EXISTING TABLE
### -------------------------------

/usr/sbin/nft delete table inet filter 2>/dev/null || true

### -------------------------------
### GENERATE RULESET
### -------------------------------

cat > "$NFT_FILE" <<EOF
#!/usr/sbin/nft -f

table inet filter {

    set $SET_WHITELIST {
        type ipv4_addr
        flags interval
    }

    set $SET_RU { 
        type ipv4_addr 
        flags interval
    }
    set $SET_CN {
        type ipv4_addr 
        flags interval 
        }
    set $SET_IN { 
        type ipv4_addr 
        flags interval 
        }

    set $SET_MANUAL {
        type ipv4_addr
        flags interval
        timeout $MANUAL_TIMEOUT
    }

    set $SET_OPTIONAL {
        type ipv4_addr
        flags interval
        timeout $MANUAL_TIMEOUT
    }

    set $SET_DOMAIN { 
        type ipv4_addr 
        flags interval 
        }

    set $SET_NOFILE {
        type ipv4_addr
        flags interval
        timeout $MANUAL_TIMEOUT
    }

    chain input {
        type filter hook input priority 0;
        policy drop;

        # Loopback
        iif lo accept

        # Whitelist always wins
        ip saddr @$SET_WHITELIST accept

        # Established traffic
        ct state established,related accept
        ct state invalid drop

        # Hard blocks
        ip saddr @$SET_MANUAL drop
        ip saddr @$SET_OPTIONAL drop
        ip saddr @$SET_NOFILE drop
        ip saddr @$SET_DOMAIN drop
        ip saddr @$SET_RU drop
        ip saddr @$SET_CN drop
        ip saddr @$SET_IN drop

        # ---- HTTP (ACME SAFE) ----
        tcp dport 80 ct state new accept

        # ---- HTTPS (PROTECTED) ----
        tcp dport 443 tcp flags syn ct state new jump https_protect

        # SSH
        tcp dport 22 ct state new accept
    }

    chain https_protect {
        # Reasonable limits for HTTPS only
        ct count over 300 drop
        limit rate over 30/second drop
        accept
    }
}
EOF

### -------------------------------
### LOAD RULESET
### -------------------------------

/usr/sbin/nft -f "$NFT_FILE"

### -------------------------------
### HELPER FUNCTIONS
### -------------------------------

load_set() {
    file="$1"
    set="$2"
    CHUNK_SIZE=500
    [ ! -s "$file" ] && return

    TMP_SORTED=$(mktemp)
    sort -Vu "$file" > "$TMP_SORTED"

    buffer=""
    count=0
    while read -r ip; do
        case "$ip" in ""|\#*) continue ;; esac
        buffer="${buffer:+$buffer, }$ip"
        count=$((count + 1))
        if [ "$count" -ge "$CHUNK_SIZE" ]; then
            nft add element "$TABLE" "$set" { $buffer } 2>/dev/null || true
            buffer=""
            count=0
        fi
    done < "$TMP_SORTED"

    [ -n "$buffer" ] && nft add element "$TABLE" "$set" { $buffer } 2>/dev/null || true
    rm -f "$TMP_SORTED"
}

load_domains() {
    [ ! -s "$BAD_DOMAINS_FILE" ] && return
    while read -r d; do
        case "$d" in ""|\#*) continue ;; esac
        dig +short "$d" | grep -E '^[0-9]' | while read -r ip; do
            nft add element "$TABLE" "$SET_DOMAIN" { $ip } 2>/dev/null || true
        done
    done < "$BAD_DOMAINS_FILE"
}

load_no_file_attempts() {
    logfile="$1"
    pattern_file="$2"
    set="$3"
    [ ! -s "$pattern_file" ] || [ ! -s "$logfile" ] && return

    while read -r pattern; do
        case "$pattern" in ""|\#*) continue ;; esac
        grep -F "$pattern" "$logfile" | \
        grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | \
        sort | uniq -c | awk '$1 >= 50 {print $2}' | \
        while read -r ip; do
            nft get element "$TABLE" "$SET_WHITELIST" { $ip } >/dev/null 2>&1 && continue
            nft add element "$TABLE" "$set" { $ip } 2>/dev/null || true
        done
    done < "$pattern_file"
}

### -------------------------------
### LOAD SETS
### -------------------------------

for ip in $WHITELIST; do
    nft add element "$TABLE" "$SET_WHITELIST" { $ip } 2>/dev/null || true
done

load_set "$SEARCH_ENGINE_WHITELIST" "$SET_WHITELIST"
load_set "$RU_FILE" "$SET_RU"
load_set "$CN_FILE" "$SET_CN"
load_set "$IN_FILE" "$SET_IN"
load_set "$MANUAL_FILE" "$SET_MANUAL"
load_set "$OPTIONAL_FILE" "$SET_OPTIONAL"
load_set "$MANDATORY_IP_FILE" "$SET_NOFILE"
load_set "$OPTIONAL_IP_FILE" "$SET_OPTIONAL"

load_domains
load_no_file_attempts "$ACCESS_LOG" "$NO_FILE_PATTERNS" "$SET_NOFILE"

rm -rf "$TMPDIR"
echo "✔ nftables firewall loaded (ACME-safe)"

firewall-incremental.sh

#!/bin/sh
set -e

TABLE="inet filter"

SET_NOFILE="no_file_block"
SET_MANUAL="manual_block"
SET_OPTIONAL="optional_block"
SET_WHITELIST="whitelist"

ACCESS_LOG="/var/log/nginx/access.log"
NO_FILE_PATTERNS="/var/log/nft/no_file.txt"
OPT_PATTERNS="/var/log/nft/opt_patterns.txt"

MANDATORY_IP_FILE="/var/log/nft/mandatory-ipv4.txt"
OPTIONAL_IP_FILE="/var/log/nft/optional-ipv4.txt"

LOG="/var/log/nft/firewall-nft-incremental.log"
LOCKFILE="/var/log/nft/firewall-nft-incremental.lock"

OPTIONAL_NSTRIKE=3

# Exit if firewall not loaded
nft list table inet filter >/dev/null 2>&1 || exit 0

touch "$MANDATORY_IP_FILE" "$OPTIONAL_IP_FILE"
chmod 600 "$MANDATORY_IP_FILE" "$OPTIONAL_IP_FILE"

exec 9>"$LOCKFILE"
flock -n 9 || exit 0

[ ! -s "$ACCESS_LOG" ] && exit 0

### Mandatory blocks
while read -r pattern; do
    case "$pattern" in ""|\#*) continue ;; esac
    grep -F "$pattern" "$ACCESS_LOG" | \
    grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -Vu | \
    while read -r ip; do
        nft get element "$TABLE" "$SET_WHITELIST" { $ip } >/dev/null 2>&1 && continue
        nft get element "$TABLE" "$SET_NOFILE" { $ip } >/dev/null 2>&1 && continue

        nft add element "$TABLE" "$SET_NOFILE" { $ip } 2>>"$LOG" || continue
        grep -qx "$ip" "$MANDATORY_IP_FILE" || echo "$ip" >> "$MANDATORY_IP_FILE"
    done
done < "$NO_FILE_PATTERNS"

### Optional blocks
[ -s "$OPT_PATTERNS" ] && \
while read -r pattern; do
    case "$pattern" in ""|\#*) continue ;; esac
    grep -F "$pattern" "$ACCESS_LOG" | \
    grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort | uniq -c | \
    awk -v n="$OPTIONAL_NSTRIKE" '$1>=n {print $2}' | \
    while read -r ip; do
        nft get element "$TABLE" "$SET_WHITELIST" { $ip } >/dev/null 2>&1 && continue
        nft get element "$TABLE" "$SET_OPTIONAL" { $ip } >/dev/null 2>&1 && continue

        nft add element "$TABLE" "$SET_OPTIONAL" { $ip } 2>>"$LOG" || continue
        grep -qx "$ip" "$OPTIONAL_IP_FILE" || echo "$ip" >> "$OPTIONAL_IP_FILE"
    done
done < "$OPT_PATTERNS"

flock -u 9

google-whitelist.sh

#!/bin/sh
set -e

### -------------------------------
### Configuration
### -------------------------------
SEARCH_ENGINE_WHITELIST="/var/log/nft/search-engine-whitelist.txt"
LOG="/var/log/nft/update-googlebot-whitelist.log"

echo "$(date -Is) | Updating Googlebot IPv4 addresses..." | tee -a "$LOG"

TMPFILE=$(mktemp)
TMPFILE_GOOGLE=$(mktemp)

### -------------------------------
### Fetch current Googlebot IPv4 only
### -------------------------------
GOOGLEBOT_JSON="https://www.gstatic.com/ipranges/goog.json"

curl -s "$GOOGLEBOT_JSON" | \
    jq -r '.prefixes[] | select(.service=="Googlebot" and has("ipv4Prefix")) | .ipv4Prefix' | \
    grep -v null > "$TMPFILE_GOOGLE"

### -------------------------------
### Keep non-Googlebot entries from existing whitelist
### -------------------------------
if [ -s "$SEARCH_ENGINE_WHITELIST" ]; then
    grep -v -f "$TMPFILE_GOOGLE" "$SEARCH_ENGINE_WHITELIST" >> "$TMPFILE"
fi

### -------------------------------
### Add latest Googlebot entries
### -------------------------------
cat "$TMPFILE_GOOGLE" >> "$TMPFILE"

### -------------------------------
### Deduplicate and save
### -------------------------------
sort -u "$TMPFILE" > "$SEARCH_ENGINE_WHITELIST"
rm -f "$TMPFILE" "$TMPFILE_GOOGLE"

COUNT=$(wc -l < "$SEARCH_ENGINE_WHITELIST")
echo "$(date -Is) | Googlebot IPv4 whitelist updated. Total entries now: $COUNT" | tee -a "$LOG"

### -------------------------------
### Optional: Reload firewall-nft.sh
### -------------------------------
# /home/admin/nft/firewall-nft.sh

Same flush and list scripts as Linux 2023

cleanup-logs.sh

#!/bin/bash
[[ -n "${BASH_VERSION:-}" ]] || { echo "This script must be run with bash"; exit 1; }

# cleanup-logs.sh
# Safe log archival + truncation with locking and retention
# Archives logs BEFORE truncation
# Retains archives ~30 days
# No disruptive service restarts

set -euo pipefail

trap 'echo "ERROR at line $LINENO: $BASH_COMMAND" >&2' ERR

LOCKFILE="/var/run/cleanup-logs.lock"
ARCHIVE_DIR="/var/log/archive"
DATE="$(date +%Y-%m-%d)"
RETENTION_DAYS=30

exec 200>"$LOCKFILE"
flock -n 200 || exit 0

mkdir -p "$ARCHIVE_DIR"

# -------------------------
# Helper functions
# -------------------------

truncate_if_exists() {
    local file="$1"
    if [[ -f "$file" ]]; then
        /usr/bin/truncate -s 0 "$file" || true
    fi
}

archive_and_truncate() {
    local src="$1"
    local name="$2"

    if [[ -f "$src" ]]; then
        cp -p "$src" "$ARCHIVE_DIR/${name}-${DATE}.log" || true
        /usr/bin/truncate -s 0 "$src" || true
    fi
}

# =========================================================
# ---- SYSTEM LOG CLEANUP (no archiving needed) ----
# =========================================================

# ---- Cloud-init Logs ----
truncate_if_exists "/var/log/cloud-init.log"
truncate_if_exists "/var/log/cloud-init-output.log"

# ---- DNF Logs ----
truncate_if_exists "/var/log/dnf.log"
truncate_if_exists "/var/log/dnf.log.1"
truncate_if_exists "/var/log/dnf.rpm.log"
truncate_if_exists "/var/log/dnf.librepo.log"
truncate_if_exists "/var/log/dnf.librepo.log.1"


# ---- Lastlog / wtmp ----
truncate_if_exists "/var/log/lastlog"
truncate_if_exists "/var/log/wtmp"

# ---- journal logs ----
journalctl --flush --rotate --vacuum-time=1s >/dev/null 2>&1 || true

# You may need old logs, so clean manually at /var/log/journal/.../system@ and user-*
# These do take up a lot of disk space

# -------------------------
# ---- APACHE2 LOGS ----
# -------------------------
NGINX_DIR="/var/log/apache2"

if [[ -d "$NGINX_DIR" ]]; then
    archive_and_truncate "$NGINX_DIR/access.log"   "access"
    archive_and_truncate "$NGINX_DIR/error.log"    "error"
    archive_and_truncate "$NGINX_DIR/access.log.1" "access.1"
    archive_and_truncate "$NGINX_DIR/error.log.1"  "error.1"

    systemctl reload nginx >/dev/null 2>&1 || true

    rm -f "$NGINX_DIR"/*.gz >/dev/null 2>&1 || true
fi

# -------------------------
# -- /var/log/*.gz files --
# -------------------------
rm -f /var/log/*.gz >/dev/null 2>&1 || true

# -------------------------
# ---- MAIL LOGS ----
# -------------------------
archive_and_truncate "/var/log/mail.log" "mail"

# -------------------------
# ---- MariaDB LOGS ----
# -------------------------
archive_and_truncate "/var/log/mariadb/mariadb.log" "mariadb"

# -------------------------
# ---- PHP-FPM LOGS ----
# -------------------------
archive_and_truncate "/var/log/php8.4-fpm.log" "php8.4-fpm-error"

# -------------------------
# ---- Archive retention ----
# -------------------------
find "$ARCHIVE_DIR" -type f -name "*.log" -mtime +"$RETENTION_DAYS" -delete || true

# ---- Finished ----
echo "Log cleanup completed at $(date)"

exit 0

nftables Security

NFTABLES & Security

chatGPT Documentation of this Solution

1️⃣ High-Level Architecture

Weekly (Full Rebuild)

Every 10 Minutes (Incremental)

2️⃣ nftables Structure Being Built

Sets Created (weekly script)

Input Chain Behavior

HTTPS Protection Chain

3️⃣ Complete File Inventory

🌍 Country Lists

🛑 Manual / Admin Control

🔎 Log-Based Detection

How They Work

Mandatory (no_file_block)

Optional (optional_block)

💾 Persisted Dynamic IP Files

🌐 Domain Blocking

🔎 Whitelist

🔐 Locking

4️⃣ What Each Script Is Doing Internally

🔵 firewall-nft.sh (Weekly Full Reset)

Step-by-step

🟢 firewall-incremental.sh (Every 10 Minutes)

Step-by-step

5️⃣ Behavioral Design Pattern

6️⃣ Important Observations

1️⃣ Timeout Behavior

2️⃣ Log Parsing Behavior

3️⃣ Possible Edge Conditions

7️⃣ Summary Diagram (Conceptual Flow)

8️⃣ Final Inventory Checklist

nft

System tools

Logs

Data directory

Install nftables

The Scripts (linux2023)

Clean Log FIles etc.

/var/log/nft files

The Scripts (Debian 13)