Securing Your Home Network: Implementation of Pangolin and Newt for Secure Internet-Facing Applications (Part 2)

hhf.technoloy · February 19, 2025, 12:26pm

In Part 1, we explored the overall architecture and set up the VPS-based Pangolin stack as our secure gateway. Now, we’ll focus on implementing the home network side - specifically creating an isolated environment for internet-exposed applications while keeping the rest of your network completely secure.

Setting Up the Isolated Network for Exposed Applications

The cornerstone of our security model is proper network isolation. We’ll implement this using a combination of VLANs, VM isolation, and Docker networks.

1. Creating Network Segments with VLANs

Most modern home routers and switches support VLANs. If yours doesn’t, consider upgrading to one that does, such as:

UniFi Dream Machine
pfSense/OPNsense on dedicated hardware
Managed switches with VLAN support

Here’s how to configure the network segments:

Management VLAN (ID: 10)

Network: 192.168.10.0/24
Gateway: 192.168.10.1
DHCP: Enabled (192.168.10.100-192.168.10.200)
Purpose: Hypervisor management, admin access

Exposed Apps VLAN (ID: 20)

Network: 192.168.20.0/24
Gateway: 192.168.20.1
DHCP: Enabled (192.168.20.100-192.168.20.200)
Purpose: VMs running Newt and internet-exposed applications

Private Apps VLAN (ID: 30)

Network: 192.168.30.0/24
Gateway: 192.168.30.1
DHCP: Enabled (192.168.30.100-192.168.30.200)
Purpose: Internal applications, never exposed to internet

Firewall Rules Implementation

Configure your router/firewall with these rules between VLANs:

# From Management (VLAN 10) to Exposed Apps (VLAN 20)
allow TCP/22 (SSH)
allow TCP/8006 (Proxmox VE)
deny all other traffic

# From Management (VLAN 10) to Private Apps (VLAN 30)
allow TCP/22 (SSH)
allow TCP/8006 (Proxmox VE)
allow TCP/80,443 (HTTP/HTTPS)
deny all other traffic

# From Exposed Apps (VLAN 20) to Management (VLAN 10)
deny all traffic

# From Exposed Apps (VLAN 20) to Private Apps (VLAN 30)
deny all traffic

# From Exposed Apps (VLAN 20) to Internet
allow all traffic

# From Private Apps (VLAN 30) to Management (VLAN 10)
deny all traffic

# From Private Apps (VLAN 30) to Exposed Apps (VLAN 20)
deny all traffic

# From Private Apps (VLAN 30) to Internet
allow HTTP/HTTPS for updates only
deny all other traffic

2. Setting Up the Proxmox Host

Configure Proxmox networking to support the VLAN setup:

Log into Proxmox web interface
Navigate to System > Network
Create VLAN interfaces:

# Management Network
Name: vmbr0.10
Type: VLAN
VLAN Tag: 10
Bridge ports: vmbr0
IP address: 192.168.10.2/24
Gateway: 192.168.10.1

# Exposed Apps Network
Name: vmbr0.20
Type: VLAN
VLAN Tag: 20 
Bridge ports: vmbr0
IP address: none (we'll assign in VM)

# Private Apps Network
Name: vmbr0.30
Type: VLAN
VLAN Tag: 30
Bridge ports: vmbr0
IP address: none (we'll assign in VM)

Apply the changes and reboot if needed.

3. Creating the Exposed Applications VM

This VM will host Newt and the applications you want to expose to the internet. It will have limited network visibility.

In Proxmox, create a new VM:

General:
  - Name: exposed-apps-vm
  - Start at boot: Yes

OS:
  - Use ISO image: ubuntu-22.04-server-amd64.iso
  - Type: Linux
  - Version: Ubuntu 22.04

System:
  - Graphics card: Default
  - SCSI Controller: VirtIO SCSI

Disks:
  - Storage: local-lvm
  - Disk size: 100 GB
  - Format: qcow2
  - Cache: writeback

CPU:
  - Sockets: 1
  - Cores: 4

Memory:
  - Memory: 8192 MB

Network:
  - Bridge: vmbr0.20 (Exposed Apps VLAN)
  - Model: VirtIO

Install Ubuntu Server 22.04 LTS following the standard installation process
Configure static IP on the VM:

Edit /etc/netplan/00-installer-config.yaml:

network:
  version: 2
  ethernets:
    ens18:
      addresses:
        - 192.168.20.10/24
      gateway4: 192.168.20.1
      nameservers:
        addresses: [1.1.1.1, 8.8.8.8]

Apply with:

sudo netplan apply

4. Setting Up Docker with Network Isolation

Install Docker and create isolated networks:

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Create dedicated networks for exposed applications
docker network create --subnet=172.20.0.0/16 exposed-apps

Implementing Newt in a Confined Environment

Newt will run in a Docker container, further isolating it from the host system and only allowing it to communicate with specific application containers.

1. Setting Up Newt Container

Create a dedicated directory for Newt:

mkdir -p /opt/newt
cd /opt/newt

Create a docker-compose.yml file:

networks:
  exposed-apps:
    external: true

services:
  newt:
    container_name: newt-client
    image: fosrl/newt:latest
    restart: unless-stopped
    environment:
      - PANGOLIN_ENDPOINT=https://pangolin.yourdomain.com
      - NEWT_ID=your_newt_id_from_pangolin
      - NEWT_SECRET=your_newt_secret_from_pangolin
    networks:
      exposed-apps:
        ipv4_address: 172.20.0.2
    cap_add:
      - NET_ADMIN
    volumes:
      - ./logs:/var/log/newt

2. Getting Configuration from Pangolin

To obtain the Newt ID and secret, you need to:

Log into your Pangolin dashboard on the VPS
Create a new organization if you don’t have one
Create a new site (select “Newt” as the connection method)
Copy the generated ID and secret
Update your docker-compose.yml with these values

3. Starting and Securing Newt

Start the Newt container:

docker compose up -d

Check that it’s running and connecting correctly:

docker logs -f newt-client

You should see messages about establishing a connection to Pangolin and creating a WireGuard interface. If everything is working, the site status in Pangolin should change to “Online”.

Connecting Exposed Applications

Now we’ll set up the applications that should be exposed to the internet. We’ll create a separate Docker Compose file for each application, ensuring they’re isolated but accessible through Newt.

1. Example: Setting Up Nextcloud

Create directory:

mkdir -p /opt/nextcloud
cd /opt/nextcloud

Create docker-compose.yml:

networks:
  exposed-apps:
    external: true
  nextcloud-internal:
    internal: true

services:
  nextcloud-db:
    image: mariadb:10.6
    container_name: nextcloud-db
    restart: unless-stopped
    command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
    networks:
      - nextcloud-internal
    volumes:
      - ./nextcloud-db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=secure_root_password
      - MYSQL_PASSWORD=secure_nextcloud_password
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
    
  nextcloud-app:
    image: nextcloud:latest
    container_name: nextcloud-app
    restart: unless-stopped
    depends_on:
      - nextcloud-db
    networks:
      nextcloud-internal:
      exposed-apps:
        ipv4_address: 172.20.0.10
    volumes:
      - ./nextcloud-data:/var/www/html
    environment:
      - MYSQL_HOST=nextcloud-db
      - MYSQL_PASSWORD=secure_nextcloud_password
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - NEXTCLOUD_TRUSTED_DOMAINS=nextcloud.yourdomain.com
      - OVERWRITEPROTOCOL=https

Start Nextcloud:

docker compose up -d

2. Example: Setting Up Jellyfin

Create directory:

mkdir -p /opt/jellyfin
cd /opt/jellyfin

Create docker-compose.yml:

networks:
  exposed-apps:
    external: true

services:
  jellyfin:
    image: jellyfin/jellyfin:latest
    container_name: jellyfin
    restart: unless-stopped
    networks:
      exposed-apps:
        ipv4_address: 172.20.0.11
    volumes:
      - ./config:/config
      - ./cache:/cache
      - /path/to/media:/media
    environment:
      - JELLYFIN_PublishedServerUrl=jellyfin.yourdomain.com

Start Jellyfin:

docker compose up -d

3. Configuring Resources in Pangolin

For each application, you need to create a resource in Pangolin:

Log into Pangolin dashboard
Go to Resources tab and click “Add Resource”
Fill in details:
- Name: Nextcloud
- Subdomain: nextcloud (will become nextcloud.yourdomain.com)
- Site: Select your site with Newt
Click “Create Resource”
On the Connectivity tab:
- Enable SSL: Yes
- Add a target:
  - Method: HTTP
  - IP Address: 172.20.0.10 (Nextcloud’s IP on the exposed-apps network)
  - Port: 80
- Click “Add Target” and “Save Changes”
On the Authentication tab:
- Configure desired authentication method
- Save changes

Repeat for each application you want to expose.

Security Monitoring and Logging

To ensure your setup remains secure, implement comprehensive monitoring:

1. Setting Up Prometheus and Grafana for Monitoring

Create a monitoring stack in your Private Apps VM:

networks:
  private-apps:
    external: true

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    networks:
      - private-apps
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
    
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    networks:
      - private-apps
    volumes:
      - grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=secure_grafana_password
      - GF_USERS_ALLOW_SIGN_UP=false

volumes:
  prometheus-data:
  grafana-data:

2. Centralizing Logs with Loki and Promtail

Add to your monitoring stack:

  loki:
    image: grafana/loki:latest
    container_name: loki
    restart: unless-stopped
    networks:
      - private-apps
    volumes:
      - ./loki:/etc/loki
      - loki-data:/loki
    command: -config.file=/etc/loki/loki-config.yml
    
  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    restart: unless-stopped
    networks:
      - private-apps
    volumes:
      - ./promtail:/etc/promtail
      - /var/log:/var/log
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: -config.file=/etc/promtail/promtail-config.yml

volumes:
  loki-data:

3. Security Alerts and Monitoring

Create Grafana dashboards for:

Newt connection status
Failed authentication attempts on Pangolin
CrowdSec alerts and actions
Network traffic patterns
Connection attempts between isolated networks

Testing the Security Boundaries

Verify that your isolation is working as expected with these tests:

1. Network Connectivity Tests

Test isolation from the Exposed Apps VM:

# Should fail - no route to Management VLAN
ping 192.168.10.1

# Should fail - no route to Private VLAN
ping 192.168.30.1

# Should succeed - internet connectivity
ping 8.8.8.8

2. Application Access Tests

Access https://nextcloud.yourdomain.com from external network
- Should prompt for authentication through Pangolin
- After auth, should access Nextcloud
Try accessing the raw IP directly:
- http://172.20.0.10 from local network
- Should fail due to network isolation
Test internal services from external network:
- Try accessing AdGuard Home or other private services
- Should be completely inaccessible

3. Security Scanning

Run security scans against your exposed domain:

# Install nmap
apt install nmap

# Scan exposed ports
nmap -sS -sV yourdomain.com

# Only ports 80, 443 and 51820 should be visible

Maintenance and Updates

Updating Pangolin Stack

Update the Pangolin stack on your VPS:

cd /opt/pangolin
docker compose pull
docker compose down
docker compose up -d

Updating Newt

Update Newt on your Exposed Apps VM:

cd /opt/newt
docker compose pull
docker compose down
docker compose up -d

Updating Applications

Each application should be updated according to its own schedule:

cd /opt/application-directory
docker compose pull
docker compose down
docker compose up -d

Benefits of This Architecture

This architecture provides several key advantages:

Zero Trust Security Model
- Every connection requires authentication
- Network segments have no implicit trust
- Applications are isolated from each other
No Exposure of Home Network
- VPS serves as the only internet-facing component
- No port forwarding on home router
- Home IP address remains private
Simplified Management
- Centralized authentication through Pangolin
- Consistent access control across all applications
- Easy addition of new exposed services
Resilience to IP Changes
- Home network can have dynamic IP
- WireGuard tunnel re-establishes automatically
- Services remain available during IP transitions
Enhanced Monitoring
- Centralized logging of access attempts
- Clear visibility of traffic patterns
- Anomaly detection across the security boundary
- Ability to quickly identify potential compromise
- Historical trends for capacity planning

Disaster Recovery Planning

Despite the security benefits of this architecture, it’s essential to plan for potential failures. Here’s how to ensure service continuity:

1. Backup Strategy

Implement a comprehensive backup strategy:

# Create backup script for Pangolin configuration
mkdir -p /opt/pangolin/backups
cat > /opt/pangolin/backup.sh << 'EOF'
#!/bin/bash
DATE=$(date +%Y-%m-%d)
BACKUP_DIR="/opt/pangolin/backups"
CONFIG_DIR="/opt/pangolin/config"

# Stop services to ensure consistent backup
cd /opt/pangolin
docker compose stop pangolin

# Create backup archive
tar -czf "$BACKUP_DIR/pangolin-config-$DATE.tar.gz" -C "$CONFIG_DIR" .

# Restart services
docker compose start pangolin

# Retain only last 7 backups
find "$BACKUP_DIR" -name "pangolin-config-*.tar.gz" -type f -mtime +7 -delete
EOF

chmod +x /opt/pangolin/backup.sh

Schedule this backup to run daily:

# Add to crontab
(crontab -l 2>/dev/null; echo "0 2 * * * /opt/pangolin/backup.sh") | crontab -

2. Failover VPS Configuration(only tested with cf)

For critical applications, consider setting up a standby VPS:

Provision a second VPS with identical configuration

Regularly sync Pangolin configuration to the standby:

rsync -avz --delete /opt/pangolin/config/ standby-vps:/opt/pangolin/config/

Configure DNS with low TTL (Time To Live) values

Create a simple health check script:

cat > /opt/health-check.sh << 'EOF'
#!/bin/bash
MAIN_VPS="pangolin.yourdomain.com"
BACKUP_VPS="standby.yourdomain.com"
DNS_API_KEY="your_dns_provider_api_key"

# Check if main VPS is responding
if curl -s --max-time 10 https://$MAIN_VPS/api/v1/health > /dev/null; then
  echo "Main VPS is healthy"
  exit 0
fi

# If we got here, main VPS is down, activate failover
echo "Main VPS appears down, activating failover..."

# Update DNS record (example using Cloudflare API)
curl -X PATCH "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \
  -H "Authorization: Bearer $DNS_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{"content":"'"$BACKUP_VPS_IP"'"}'

echo "Failover complete. DNS should propagate shortly."
EOF

chmod +x /opt/health-check.sh

3. Recovery Documentation

Create detailed recovery documentation that includes:

Complete network diagram with all IP addresses and VLANs
Step-by-step recovery instructions for each component
Configuration backup locations and restoration procedures
Contact information for service providers and domain registrars
Credentials stored in a secure password manager

Advanced Network Hardening

To further strengthen your setup, consider these advanced hardening techniques:

1. Implement Network Intrusion Detection (Use at your own desecration)

Deploy Suricata in IDS mode on your router/firewall to detect suspicious activities:

# Install Suricata on a dedicated monitoring system or security gateway
apt install suricata

# Configure Suricata to monitor traffic between VLANs
cat > /etc/suricata/suricata.yaml << 'EOF'
# Suricata configuration snippet
...
af-packet:
  - interface: eth0.20
    cluster-id: 99
    cluster-type: cluster_flow
    defrag: yes
  - interface: eth0.30
    cluster-id: 98
    cluster-type: cluster_flow
    defrag: yes
...
EOF

# Enable and start Suricata
systemctl enable suricata
systemctl start suricata

2. Implement MAC Address Filtering (Use at your own desecration)

For an additional layer of security, implement MAC address filtering on your router/switch to only allow known devices on each VLAN:

# Example configuration for UniFi Security Gateway
config firewall rule {
    name 'VLAN20 MAC Filter'
    description 'Only allow authorized devices on Exposed Apps VLAN'
    default-action drop
    rule {
        action accept
        description 'Allow Exposed Apps VM'
        mac-address AA:BB:CC:DD:EE:FF
    }
    interface eth0.20
}

3. Regular Security Auditing

Implement regular security audits:

# Create security audit script
cat > /opt/security-audit.sh << 'EOF'
#!/bin/bash
DATE=$(date +%Y-%m-%d)
REPORT_DIR="/opt/security/reports"
mkdir -p $REPORT_DIR

echo "Starting security audit on $DATE" > "$REPORT_DIR/audit-$DATE.log"

# Check for unauthorized containers on exposed network
echo "=== Exposed network containers ===" >> "$REPORT_DIR/audit-$DATE.log"
docker network inspect exposed-apps | grep -E '"Name"|"IPv4Address"' >> "$REPORT_DIR/audit-$DATE.log"

# Check for unauthorized connections
echo "=== Unexpected connections ===" >> "$REPORT_DIR/audit-$DATE.log"
netstat -tnp | grep -v -E '(127.0.0.1|172.20.0)' >> "$REPORT_DIR/audit-$DATE.log"

# Check Docker image vulnerabilities
echo "=== Container vulnerabilities ===" >> "$REPORT_DIR/audit-$DATE.log"
for container in $(docker ps --format '{{.Image}}'); do
  trivy image $container --severity HIGH,CRITICAL >> "$REPORT_DIR/audit-$DATE.log"
done

# Check for failed authentication attempts
echo "=== Failed authentications ===" >> "$REPORT_DIR/audit-$DATE.log"
grep "authentication failure" /var/log/auth.log | tail -100 >> "$REPORT_DIR/audit-$DATE.log"

echo "Security audit complete. Report saved to $REPORT_DIR/audit-$DATE.log"
EOF

chmod +x /opt/security-audit.sh

Schedule weekly audits:

(crontab -l 2>/dev/null; echo "0 3 * * 0 /opt/security-audit.sh") | crontab -

Scaling the Architecture

As your self-hosting needs grow, you may need to scale this architecture. Here’s how to do it seamlessly:

1. Multiple Application Groups

You can create multiple isolated application groups by replicating the pattern:

# Create additional isolated networks
docker network create --subnet=172.21.0.0/16 media-apps
docker network create --subnet=172.22.0.0/16 productivity-apps
docker network create --subnet=172.23.0.0/16 game-servers

Each network would have its own Newt client with specific access restrictions:

# Example media-apps Newt client
networks:
  media-apps:
    external: true
services:
  media-newt:
    container_name: media-newt-client
    image: fosrl/newt:latest
    restart: unless-stopped
    environment:
      - PANGOLIN_ENDPOINT=https://pangolin.yourdomain.com
      - NEWT_ID=media_newt_id_from_pangolin
      - NEWT_SECRET=media_newt_secret_from_pangolin
    networks:
      media-apps:
        ipv4_address: 172.21.0.2
    cap_add:
      - NET_ADMIN

2. Multiple Physical Locations

If you have applications in multiple physical locations, you can set up Newt clients at each location:

Create a site in Pangolin for each location
Deploy Newt to each location
Configure resources to use the appropriate site
Maintain separate network isolation at each location

Performance Optimization

While security is the primary focus, performance optimization ensures a good user experience:

1. Content Delivery Optimization

Implement Traefik cache middleware on your Pangolin VPS:

# Add to Traefik configuration
http:
  middlewares:
    cache:
      plugin:
        souin:
          default_cache:
            ttl: "3600s"
          urls:
            "static.yourdomain.com":
              ttl: "86400s"
            "media.yourdomain.com/thumbnails":
              ttl: "43200s"
          default_cache_control: "public, max-age=86400"
          log_level: "info"

2. Application-Specific Optimizations

For media streaming applications like Jellyfin, optimize for direct streaming:

# Jellyfin container configuration
environment:
  - JELLYFIN_FFmpegProbeSize=10000000
  - JELLYFIN_TranscodingTempPath=/transcode
  - JELLYFIN_EnableHardwareEncoding=true
volumes:
  - /dev/dri:/dev/dri # For hardware acceleration
  - ./transcode:/transcode
  - ./config:/config
  - ./media:/media:ro

Troubleshooting Common Issues

Prepare for common issues by documenting troubleshooting steps:

1. Tunnel Connectivity Issues

If Newt loses connection to Pangolin:

# Check Newt logs
docker logs newt-client

# Check route table
ip route

# Test connection to Pangolin
curl -v --max-time 10 https://pangolin.yourdomain.com/api/v1/health

# Restart Newt if needed
docker compose restart newt

2. Application Access Issues

If applications are inaccessible:

# Verify application container is running
docker ps | grep application-name

# Check application logs
docker logs application-container

# Verify network connectivity from Newt to application
docker exec -it newt-client ping 172.20.0.10

# Check Pangolin resource configuration
curl -v --max-time 10 https://pangolin.yourdomain.com/api/v1/resources/application-name

3. Authentication Issues

If users can’t authenticate:

# Check Pangolin authentication logs
docker logs pangolin | grep "authentication"

# Verify Traefik is forwarding authentication headers
docker logs traefik | grep "forward auth"

# Test authentication flow
curl -v --max-time 10 https://pangolin.yourdomain.com/api/v1/auth/check

# Restart authentication components if needed
docker compose restart pangolin traefik

Future Enhancements

Consider these future enhancements to further improve your setup:

1. Geo-Location Based Access Control

Implement country-based restrictions:

# Add GeoIP middleware to Traefik
http:
  middlewares:
    country-whitelist:
      ipWhiteList:
        sourceRange:
          - "10.0.0.0/8" # Local networks
        ipStrategy:
          depth: 1
        geoip:
          countries:
            - "US"
            - "CA"

3. Behavioral Analytics

Implement advanced threat detection using behavioral analytics:

# Deploy Elastic Stack for behavioral analytics
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.7.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=secure_elastic_password
  
  kibana:
    image: docker.elastic.co/kibana/kibana:8.7.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
  
  elastic-agent:
    image: docker.elastic.co/beats/elastic-agent:8.7.0
    environment:
      - FLEET_ENROLLMENT_TOKEN=your_fleet_token
      - FLEET_URL=https://fleet.yourdomain.com
      - FLEET_INSECURE=false

Conclusion

The multi-part architecture we’ve designed creates a robust, secure environment for hosting internet-accessible applications while maintaining strong isolation principles. By placing Pangolin on a VPS and running Newt in an isolated environment on your local network, you’ve created a security model with multiple layers of protection.

Key benefits of this implementation include:

Complete network isolation between internet-exposed and private applications
Zero-trust security model where all access requires explicit authentication
No direct internet exposure of your home network or private services
Centralized management through the Pangolin dashboard
Flexible scaling options for future growth
Resilience to network changes through WireGuard’s persistent tunnels
Comprehensive monitoring and security controls

By following this architecture, you’ve created a setup that rivals commercial solutions while maintaining complete control over your data and infrastructure. As self-hosting becomes increasingly popular, this approach provides the perfect balance between accessibility and security.

Remember that security is an ongoing process. Regularly update all components, review logs, conduct security audits, and stay informed about emerging threats to ensure your self-hosted infrastructure remains safe and reliable.