Deep Dive into System Administration : Proxmox, Firewalls, Encrypted storage, etc.
Beyond the Basics
The Challenge: Take Proxmox from "it works" to "production-ready." Advanced networking with nftables. Nested containers running Docker. Full-disk encryption with LUKS. Automated container deployment via custom scripts.
This isn't a tutorial you follow. This is infrastructure you architect.
The goal? Build something that would actually survive in a real datacenter.
After mastering the basics of Proxmox (containers, VMs, basic networking), I wanted to push deeper. This TP covered everything from advanced firewall rules to encryption, from nested virtualization to automation scripting.
Here's what I builtโand more importantly, why it matters.
Part 1: Advanced Firewall with nftables
The Problem: Default Proxmox firewall is fine for basic scenarios, but it's limited. nftables offers granular control, better performance, and modern syntax.
๐ฅ Why nftables over iptables?
| Feature | iptables (old) | nftables (modern) |
|---|---|---|
| Syntax | Complex, cryptic | Clean, readable |
| Performance | Rule-by-rule evaluation | Optimized rulesets |
| IPv4/IPv6 | Separate commands | Unified framework |
| Atomicity | No | Yes (atomic updates) |
โ๏ธ nftables Configuration
Created a comprehensive ruleset for the Proxmox host:
# /etc/nftables.conf
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
chain input {
type filter hook input priority 0; policy drop;
# Allow established/related connections
ct state established,related accept
# Allow loopback
iif lo accept
# Allow ICMP (ping)
ip protocol icmp accept
ip6 nexthdr icmpv6 accept
# Allow SSH (rate limited)
tcp dport 22 ct state new limit rate 3/minute accept
# Allow Proxmox Web UI
tcp dport 8006 accept
# Log and drop everything else
log prefix "nftables-drop: " drop
}
chain forward {
type filter hook forward priority 0; policy accept;
# Allow container traffic
ct state established,related accept
}
chain output {
type filter hook output priority 0; policy accept;
}
}
๐ NAT Rules for Container Network
Separate NAT configuration file for cleaner management:
# /etc/nftables.d/nat_rules.conf
table ip nat {
chain prerouting {
type nat hook prerouting priority -100;
# Port forwarding example: HTTP to container
iif "vmbr0" tcp dport 80 dnat to 10.0.0.100:80
# HTTPS to container
iif "vmbr0" tcp dport 443 dnat to 10.0.0.100:443
}
chain postrouting {
type nat hook postrouting priority 100;
# Masquerade outgoing traffic from containers
oif "vmbr0" masquerade
}
}
# Enable and test nftables
systemctl enable nftables
systemctl start nftables
# Verify rules
nft list ruleset
# Include NAT rules in main config
echo 'include "/etc/nftables.d/nat_rules.conf"' >> /etc/nftables.conf
nft -f /etc/nftables.conf
Pro Tip: Always test firewall rules on a console session (not SSH) before applying. Lock yourself out once, you'll never forget this advice.
Part 2: Storage - Mount Points & Partitions
Proxmox uses storage pools, but understanding underlying Linux storage is crucial.
๐พ Creating a Mount Point
# List available disks
lsblk
fdisk -l
# Create partition (using fdisk or parted)
fdisk /dev/sdb
# n (new partition)
# p (primary)
# 1 (partition number)
# [default start]
# [default end - use full disk]
# w (write changes)
# Format partition as ext4
mkfs.ext4 /dev/sdb1
# Create mount point
mkdir -p /mnt/data
# Mount partition
mount /dev/sdb1 /mnt/data
# Make mount persistent (add to /etc/fstab)
echo '/dev/sdb1 /mnt/data ext4 defaults 0 2' >> /etc/fstab
# Verify
df -h | grep /mnt/data
๐ LUKS Encryption - Full Disk Encryption
For sensitive data, encryption isn't optional. LUKS (Linux Unified Key Setup) provides transparent disk encryption.
โ ๏ธ Warning: Encrypting a partition DESTROYS all existing data. Backup first!
# Install cryptsetup
apt install cryptsetup -y
# Create LUKS encrypted partition
cryptsetup luksFormat /dev/sdb1
# Type 'YES' (uppercase)
# Enter passphrase (STRONG password!)
# Open encrypted partition
cryptsetup luksOpen /dev/sdb1 encrypted_data
# Enter passphrase
# Format the encrypted partition
mkfs.ext4 /dev/mapper/encrypted_data
# Mount encrypted partition
mkdir -p /mnt/encrypted
mount /dev/mapper/encrypted_data /mnt/encrypted
# Verify encryption
cryptsetup status encrypted_data
df -h | grep encrypted
๐ Auto-unlock on Boot (Optional but Convenient)
# Create keyfile (more secure than password on boot)
dd if=/dev/urandom of=/root/luks-keyfile bs=512 count=4
chmod 400 /root/luks-keyfile
# Add keyfile to LUKS
cryptsetup luksAddKey /dev/sdb1 /root/luks-keyfile
# Configure crypttab for auto-unlock
echo 'encrypted_data /dev/sdb1 /root/luks-keyfile luks' >> /etc/crypttab
# Add to fstab
echo '/dev/mapper/encrypted_data /mnt/encrypted ext4 defaults 0 2' >> /etc/fstab
Pro Tip: Store the LUKS header backup in a safe place. If the header corrupts, your data is gone forever.
# Backup LUKS header
cryptsetup luksHeaderBackup /dev/sdb1 --header-backup-file /root/luks-header-backup-sdb1.img
Part 3: Containers vs VMs - Understanding the Difference
Proxmox supports both LXC containers and KVM virtual machines. Knowing when to use each is critical.
| Feature | LXC Container | KVM Virtual Machine |
|---|---|---|
| Isolation | Process-level (shared kernel) | Hardware-level (full isolation) |
| Performance | Near-native (very fast) | Overhead from virtualization |
| Boot Time | Seconds | Minutes |
| Resource Usage | Minimal (MBs of RAM) | Higher (GBs of RAM) |
| OS Support | Linux only | Any OS (Windows, BSD, etc.) |
| Use Case | Web servers, databases, services | Different OS, legacy apps, testing |
Key Insight: Containers are NOT just "lightweight VMs." They're fundamentally different. Containers share the host kernelโfast but less isolated. VMs have their own kernelโslower but fully independent.
Part 4: Nested Containers - Docker inside LXC
Running Docker inside a Proxmox LXC container (nested virtualization) unlocks powerful possibilities. Perfect for self-hosted apps like Immich (photo management).
๐ณ Setting Up Docker in LXC
โ ๏ธ Important: Not all LXC templates support nesting. Must use an unprivileged container with nesting enabled.
Create Container with Nesting
In Proxmox UI: Create CT โ Features tab โ Check "Nesting"
# Or via CLI
pct create 200 local:vztmpl/debian-12-standard_12.0-1_amd64.tar.zst \
--hostname docker-host \
--memory 4096 \
--net0 name=eth0,bridge=vmbr0,ip=dhcp \
--features nesting=1 \
--unprivileged 1
Install Docker
# Inside the container
apt update && apt upgrade -y
apt install curl -y
# Install Docker using official script
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh
# Install Docker Compose
apt install docker-compose -y
# Verify installation
docker --version
docker-compose --version
Deploy Immich with Docker Compose
Immich is a self-hosted Google Photos alternative. Perfect for testing Docker deployments.
# Create project directory
mkdir -p /opt/immich
cd /opt/immich
# Download Immich docker-compose
wget -O docker-compose.yml https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
wget -O .env https://github.com/immich-app/immich/releases/latest/download/example.env
# Edit .env file (set upload location, timezone, etc.)
nano .env
# Deploy Immich
docker-compose up -d
# Check status
docker-compose ps
docker-compose logs -f
Why Nested Containers? Isolation + Portability. The LXC container isolates Immich from the host. Docker Compose makes the entire stack portable. Backup the container, restore anywhere.
๐ธ Immich Setup
After deployment, access Immich at http://CONTAINER_IP:2283. Create admin account, start uploading photos. Full Google Photos alternative running on your infrastructure.
Automatic photo backup from iOS/Android. Set it and forget itโjust like Google Photos.
Machine learning for object/face recognition. Search "beach" or "dog" and find relevant photos.
View photos on a map based on GPS metadata. Perfect for travel memories.
Your photos stay on YOUR server. No cloud scanning. No data mining. Full control.
Part 5: Automation - CT-Creator Script
Manual container creation is tedious. Automation is essential for scale. Enter: CT-Creatorโmy Bash script for automated Proxmox container deployment.
CT-Creator
Automated LXC container creation script for Proxmox. Configure once, deploy instantly.
View on GitHub๐ What CT-Creator Does
Prompts for container ID, hostname, RAM, disk size, network config. No manual CLI syntax.
Choose from available OS templates (Debian, Ubuntu, etc.). Automatic download if missing.
DHCP or static IP. Auto-bridge selection. Firewall integration.
Auto-start container, run initial updates, install essential packages.
๐ป Script Highlights
# Key features from CT-Creator
#!/bin/bash
# Validation function - ensures no duplicate CT IDs
validate_ctid() {
if pct status "$1" &>/dev/null; then
echo "Error: Container ID $1 already exists!"
exit 1
fi
}
# Auto-detect available templates
list_templates() {
pveam update
pveam available | grep -E "(debian|ubuntu|alpine)" | awk '{print $2}'
}
# Create container with error handling
create_container() {
pct create "$CTID" "$TEMPLATE" \
--hostname "$HOSTNAME" \
--memory "$RAM" \
--swap "$SWAP" \
--disk "$DISK" \
--net0 name=eth0,bridge="$BRIDGE",ip="$IP" \
--features nesting="$NESTING" \
--unprivileged "$UNPRIVILEGED" \
--password || {
echo "Container creation failed!"
exit 1
}
}
# Post-creation tasks
post_setup() {
pct start "$CTID"
sleep 5
pct exec "$CTID" -- bash -c "apt update && apt upgrade -y"
echo "Container $CTID ($HOSTNAME) ready!"
}
๐ Usage Example
# Clone the repository
git clone https://github.com/TristanDivanach/CT-creator.git
cd CT-creator
# Make script executable
chmod +x ct-creator.sh
# Run interactive mode
./ct-creator.sh
# Follow prompts:
# - Enter CT ID: 300
# - Hostname: webserver
# - RAM (MB): 2048
# - Disk (GB): 20
# - Template: debian-12
# - Network: DHCP or Static
# - Enable nesting: yes/no
# Script handles the rest!
Why Automation Matters: Deploying 10 containers manually = 30 minutes + errors. With CT-Creator = 2 minutes, zero errors. Reproducible infrastructure.
Integration: Putting It All Together
Here's how all these components work together in a real infrastructure:
Protects Proxmox host. Rate-limits SSH. Forwards ports to containers. NAT for outbound traffic.
LUKS-encrypted partition stores sensitive data. Immich photos, backups, database dumpsโall encrypted at rest.
Lightweight, fast, isolated environments. Each service gets its own container. Easy to backup/restore.
Docker inside LXC for complex apps. Immich runs as Docker Compose stack. Portable and maintainable.
Deploy new containers in seconds. Consistent configuration. Version-controlled via Git.
Separate data storage from system. Easy to expand. Bind mounts into containers for shared storage.
Lessons Learned
Firewall + encryption + isolation. One layer fails, others protect. Defense in depth isn't paranoiaโit's professionalism.
Spent 2 hours writing CT-Creator. Saved 20+ hours on deployments. Good docs are force multipliers.
Separate concerns. Firewall rules in one file. NAT in another. Storage isolated. Easier to maintain and troubleshoot.
Every manual task automated is time saved forever. Scripts improve over time. Manual work doesn't.
Don't just copy commands. Know WHY nftables > iptables. WHY LUKS matters. WHY containers โ VMs. Knowledge stacks.
Proxmox snapshots are lifesavers. Test destructive changes (firewall rules, encryption) on snapshots first. Learn without fear.
Mistakes I Made (And How I Fixed Them)
Mistake #1: Applied nftables rules without testing SSH access first โ Locked out
Fix: Used Proxmox web console. Added SSH rule BEFORE enabling firewall. Lesson learned.
Mistake #2: Forgot to enable nesting on LXC โ Docker refused to start
Fix: Destroyed container, recreated with --features nesting=1. Now part of CT-Creator checklist.
Mistake #3: Encrypted partition without backing up LUKS header โ Almost lost everything during testing
Fix: Fortunately it was test data. Now ALWAYS backup header: cryptsetup luksHeaderBackup
Mistake #4: Hard-coded values in CT-Creator script โ Broke when storage pool changed
Fix: Made script fully interactive. Auto-detects available storage, templates, bridges. Much more robust.
What's Next?
This infrastructure is solid, but there's always room to grow:
Prometheus + Grafana for metrics. Loki for log aggregation. Alert on resource exhaustion.
Proxmox Backup Server (PBS) integration. Scheduled snapshots. Offsite replication.
Nginx Proxy Manager or Traefik. SSL certificates via Let's Encrypt. Subdomain routing.
Authelia or Keycloak for centralized authentication. One login for all services.
The Bottom Line
This wasn't just a TP. This was a deep dive into production-grade infrastructure.
- nftables: Modern firewall with granular control
- LUKS: Enterprise-level encryption for data at rest
- Nested containers: Docker inside LXC for complex deployments
- Automation: Scripts that save hours and eliminate errors
- Real applications: Immich running on self-hosted infrastructure
Every component serves a purpose. Every decision has a rationale. This is what real system administration looks like.
Build infrastructure that matters.
Automate what's repetitive.
Learn by doing, not just reading.
Want to Try CT-Creator?
The script is open source. Clone it, modify it, make it yours.
Star the repo if you find it useful. PRs welcome!
Questions? Suggestions? Improvements for CT-Creator? Open an issue on GitHub or drop a comment below.
Infrastructure is code. Code is power. ๐ป