Setting up automated Pangolin monitoring with systemd

[hhf.technoloy]

Pangolin Monitoring Setup Guide

This guide explains how to set up comprehensive monitoring for your Pangolin server using a combination of systemd services, Discord alerts, and optional Falco security monitoring.

Basic Monitoring Setup

1. Install and Configure the Monitor Script

First, create the monitoring directory and download the script:

sudo mkdir -p /opt/pangolin-monitor
curl -o /opt/pangolin-monitor/pangolin-monitor.sh https://gist.githubusercontent.com/hhftechnology/b2109fa3f2bf05ebb8945f0423d9e6b9/raw/58514da761cdb53bbd8144c71999347cd2a8f1ea/pangolin-monitor.sh

Before making the script executable, you need to edit it to configure your specific settings:

sudo nano /opt/pangolin-monitor/pangolin-monitor.sh

Update the following configuration variables at the top of the script:

# Configuration Section
PANGOLIN_URL="https://pangolin.testing.your.domain"  # Your Pangolin instance URL
PANGOLIN_EMAIL="your_email@hhf.technology"          # Your admin email
PANGOLIN_PASSWORD="your_admin_password"             # Your admin password
PANGOLIN_ORG="your_org"                            # Your organization name
DISCORD_WEBHOOK="https://discord.com/api/webhooks/your-hook"  # Your Discord webhook URL

# Optional: Adjust monitoring thresholds if needed
CHECK_INTERVAL=60                    # How often to check (in seconds)
BANDWIDTH_WARNING_THRESHOLD=1000     # MB
BANDWIDTH_CRITICAL_THRESHOLD=2000    # MB

After configuring the script, make it executable:

sudo chmod +x /opt/pangolin-monitor/pangolin-monitor.sh

Important Configuration Notes:

1. PANGOLIN_URL: Must be the full URL to your Pangolin instance
2. PANGOLIN_EMAIL and PANGOLIN_PASSWORD: Must match your admin credentials
3. PANGOLIN_ORG: The organization name in Pangolin you want to monitor
4. DISCORD_WEBHOOK: Create this in your Discord server:
   • Open Discord Server Settings
   • Go to Integrations
   • Create a Webhook
   • Copy the Webhook URL

Adjusting Thresholds:

• CHECK_INTERVAL: Default 60 seconds, increase for less frequent checks
• BANDWIDTH_WARNING_THRESHOLD: Default 1000 MB
• BANDWIDTH_CRITICAL_THRESHOLD: Default 2000 MB
• Adjust these based on your typical usage patterns

2. Create Systemd Service

Create the service file at /etc/systemd/system/pangolin-monitor.service:

sudo nano /etc/systemd/system/pangolin-monitor.service

Add the following content:

[Unit]
Description=Pangolin Monitoring Service
After=network.target

[Service]
Type=simple
ExecStart=/opt/pangolin-monitor/pangolin-monitor.sh
Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target

3. Enable and Start the Service

sudo systemctl enable pangolin-monitor.service
sudo systemctl start pangolin-monitor.service

Monitoring Features

The monitoring script provides comprehensive checks including:

Container Health Monitoring

• Checks status of all critical containers:
  • Pangolin
  • Gerbil
  • Traefik
• Verifies both running state and health check status
• Sends alerts for any container issues

Bandwidth Monitoring

• Tracks total bytes in/out
• Configurable warning and critical thresholds
• Bandwidth metrics in logs and alerts
• Uses Pangolin API for metrics collection

Enhanced Security with Falco

For additional security monitoring, you can deploy Falco:

sudo docker run --rm -i -t --name falco --privileged \
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    -v /etc:/host/etc:ro \
    falcosecurity/falco:0.39.2

Falco provides:

• Real-time threat detection
• Container runtime monitoring
• Kubernetes workload security
• Behavioral monitoring

Security Recommendations

1. SSH Security:

   • Avoid exposing port 22 to the public internet
   • Use key-based authentication only
   • Consider using a jump host or VPN for SSH access

2. Container Security:

   • Keep containers updated
   • Use minimal base images
   • Implement resource limits
   • Regular security scanning

3. Network Security:

   • Implement proper firewall rules
   • Use reverse proxy with SSL
   • Regular security audits
   • Monitor unusual traffic patterns

Monitoring Verification

Check Service Status

sudo systemctl status pangolin-monitor.service

View Monitoring Logs

journalctl -u pangolin-monitor.service

Troubleshooting

Common Issues

1. Discord Alerts Not Working

   • Verify webhook URL in script
   • Check network connectivity
   • Verify Discord channel permissions

2. Container Status Errors

   • Check Docker logs
   • Verify container health checks
   • Check resource utilization

3. Bandwidth Monitoring Issues

   • Verify API access
   • Check threshold configurations
   • Verify metrics collection

Additional Resources

• Full discussion and updates: HHF Technology Forum
• Detailed blog post: Pangolin Monitor Blog
• Falco Documentation

Customization

The monitoring script can be customized by editing thresholds and alert conditions in /opt/pangolin-monitor/pangolin-monitor.sh. Common customizations include:

• Adjusting bandwidth thresholds
• Modifying check intervals
• Adding custom metrics
• Configuring additional alert destinations

Remember to restart the service after making changes:

sudo systemctl restart pangolin-monitor.service

[player684]

Hi, This is really helpful. Thanks for this.

If I have 2FA on Pangolin account would this still work because it need Pangolin password and with 2FA I guess password alone won’t be enough.

[hhf.technoloy]

glad to hear this. yes it will work , you can put up discord alerts too.

[player684]

yeah that’s what i plan to do with this without the Falco bit for now because I don’t understand Flaco and want to be careful not to block myself

[hhf.technoloy]

you can secure 22 by tailnet. I can share my script if you wish

[player684]

Please do. But I am using a custom port and I am also mindful not to mess-up Pangolin
Would you recommend Falco or Crowdsec or both? What is the best setup in your opinion?

[hhf.technoloy]

bash script that securely binds SSH to your Tailscale IP

#!/bin/bash

# Exit on any error and enable error tracing
set -e
set -x

# Function to log messages with timestamps
log() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1"
}

# Function to check command status and exit if failed
check_status() {
    if [ $? -eq 0 ]; then
        log "✅ $1 completed successfully"
    else
        log "❌ Error: $1 failed"
        exit 1
    fi
}

# Function to verify IPv4 forwarding is enabled
check_ipv4_forwarding() {
    if [ "$(cat /proc/sys/net/ipv4/ip_forward)" -eq 1 ]; then
        log "IPv4 forwarding is enabled"
        return 0
    else
        log "IPv4 forwarding is not enabled"
        return 1
    fi
}

# Function to create secure user and set up SSH keys
setup_secure_user() {
    local username="secureuser"
    local ssh_dir="/home/$username/.ssh"
    
    # Create user if doesn't exist
    if ! id "$username" &>/dev/null; then
        log "Creating secure user..."
        useradd -m -s /bin/bash "$username"
        # Set a secure random password that will be disabled after key setup
        local temp_pass=$(openssl rand -base64 32)
        echo "$username:$temp_pass" | chpasswd
        
        # Add user to sudo group and configure sudo access
        usermod -aG sudo "$username"
        # Configure passwordless sudo for specific commands
        echo "$username ALL=(ALL:ALL) NOPASSWD: /usr/sbin/tailscale, /bin/systemctl restart ssh, /bin/systemctl restart sshd" > "/etc/sudoers.d/$username"
        chmod 0440 "/etc/sudoers.d/$username"
    fi
    
    # Create and secure SSH directory
    mkdir -p "$ssh_dir"
    chown -R "$username:$username" "$ssh_dir"
    chmod 700 "$ssh_dir"
    
    # Generate SSH key pair
    log "Generating SSH key pair..."
    local key_file="$ssh_dir/id_ed25519"
    if [ ! -f "$key_file" ]; then
        sudo -u "$username" ssh-keygen -t ed25519 -f "$key_file" -N "" -C "$username@$(hostname)"
        cat "$key_file.pub" >> "$ssh_dir/authorized_keys"
        chmod 600 "$ssh_dir/authorized_keys"
        chown "$username:$username" "$ssh_dir/authorized_keys"
    fi
    
    # Disable password authentication for this user
    passwd -l "$username"
    
    # Display the private key for backup and wait for user confirmation
    log "⚠️ IMPORTANT - SSH Private Key ⚠️"
    log "Please copy this private key and store it securely. You will need it to connect to this server."
    log "The key will be displayed below. Once displayed, you will have time to copy it."
    echo ""
    read -p "Press Enter to display the private key..."
    echo ""
    echo "------------------------"
    echo "PRIVATE KEY - COPY EVERYTHING BETWEEN THE LINES:"
    echo "------------------------"
    cat "$key_file"
    echo "------------------------"
    echo ""
    
    # Wait for user confirmation with a clear warning
    while true; do
        read -p "Have you securely copied the private key? (yes/no): " confirm
        case $confirm in
            [Yy]* ) break;;
            [Nn]* ) 
                log "Please copy the key now. It's critical for server access."
                echo "------------------------"
                cat "$key_file"
                echo "------------------------"
                echo "";;
            * ) log "Please answer yes or no.";;
        esac
    done
    
    # Display public key for reference
    log "For reference, here is the public key (less critical, but useful to keep):"
    echo "------------------------"
    cat "$key_file.pub"
    echo "------------------------"
    echo ""
    read -p "Press Enter to continue with the setup..."
}

# Function to determine SSH service name
get_ssh_service() {
    if systemctl list-units --full -all | grep -Fq "ssh.service"; then
        echo "ssh"
    elif systemctl list-units --full -all | grep -Fq "sshd.service"; then
        echo "sshd"
    else
        log "SSH service not found!"
        exit 1
    fi
}

# Check if script is run as root
if [ "$EUID" -ne 0 ]; then 
    log "Please run as root"
    exit 1
fi

# 1. Set up secure user and generate SSH keys
log "Setting up secure user and SSH keys..."
setup_secure_user
check_status "Secure user setup"

# 2. Verify Tailscale installation
log "Checking Tailscale installation..."
if ! command -v tailscale &> /dev/null; then
    log "Tailscale is not installed. Please install Tailscale first."
    exit 1
fi
check_status "Tailscale installation check"

# 2. Verify Tailscale status
log "Checking Tailscale status..."
if ! tailscale status &> /dev/null; then
    log "Tailscale is not running"
    exit 1
fi
check_status "Tailscale status check"

# 3. Check IPv4 forwarding
log "Checking IPv4 forwarding..."
if ! check_ipv4_forwarding; then
    log "Enabling IPv4 forwarding..."
    echo 1 > /proc/sys/net/ipv4/ip_forward
    # Make IPv4 forwarding persistent
    echo "net.ipv4.ip_forward = 1" > /etc/sysctl.d/99-tailscale.conf
    sysctl -p /etc/sysctl.d/99-tailscale.conf
fi
check_status "IPv4 forwarding configuration"

# 4. Get Tailscale IP
log "Getting Tailscale IP..."
TAILSCALE_IP=$(tailscale ip -4)
if [ -z "$TAILSCALE_IP" ]; then
    log "Failed to get Tailscale IP"
    exit 1
fi
log "Tailscale IP: $TAILSCALE_IP"
check_status "Tailscale IP retrieval"

# 5. Backup original SSH configuration
log "Creating backup of SSH configuration..."
cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup.$(date +%Y%m%d_%H%M%S)
check_status "SSH config backup"

# 6. Update SSH configuration
log "Updating SSH configuration..."
cat > /etc/ssh/sshd_config << EOF
# SSH Configuration - Modified by Tailscale binding script
Port 22
ListenAddress ${TAILSCALE_IP}
PermitRootLogin prohibit-password
PasswordAuthentication no
PubkeyAuthentication yes
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server

# Security enhancements
Protocol 2
PermitEmptyPasswords no
MaxAuthTries 3
ClientAliveInterval 300
ClientAliveCountMax 2
EOF
check_status "SSH config update"

# 7. Verify SSH configuration syntax
log "Verifying SSH configuration syntax..."
sshd -t
check_status "SSH config syntax verification"

# 8. Restart SSH service
SSH_SERVICE=$(get_ssh_service)
log "Restarting ${SSH_SERVICE} service..."
systemctl restart ${SSH_SERVICE}
check_status "SSH service restart"

# 9. Verify SSH service is running
log "Verifying SSH service status..."
if ! systemctl is-active --quiet ${SSH_SERVICE}; then
    log "SSH service failed to start. Rolling back changes..."
    cp /etc/ssh/sshd_config.backup.$(date +%Y%m%d_%H%M%S) /etc/ssh/sshd_config
    systemctl restart ${SSH_SERVICE}
    exit 1
fi
check_status "SSH service status verification"

log "✨ Setup completed successfully! ✨"
log "Your SSH service is now bound to Tailscale IP: ${TAILSCALE_IP}"
log ""
log "⚠️  Important Notes:"
log "1. Keep this terminal session open until you verify SSH access"
log "2. Test SSH connection using: ssh user@${TAILSCALE_IP}"
log "3. To revert changes:"
log "   - cp /etc/ssh/sshd_config.backup.$(date +%Y%m%d_%H%M%S) /etc/ssh/sshd_config"
log "   - systemctl restart ${SSH_SERVICE}"

Here’s a more detailed step-by-step procedure:

SSH Key Setup and User Creation Procedure

1. Save the Script

   • Create a new file named bind-ssh-tailscale.sh
   • Copy the provided script into this file
   • Save it in a location you can easily access

2. Set Script Permissions

   chmod +x bind-ssh-tailscale.sh
   

3. Execute the Script

   sudo ./bind-ssh-tailscale.sh
   

Note: Keep the terminal window open during the entire process. donot not close it

4. Secure the Generated Key

   • When the script runs, it will display a private SSH key
   • Copy the entire key output, including the header and footer lines
   • Create a new file with .key extension (e.g., secure_access.key)
   • Paste the copied key into this file
   • Save the file in a secure location

5. Convert Key for PuTTY Use

   • Open PuTTYgen on your Windows system
   • Click “Load” and select your .key file
     • Make sure to select “All Files (.)” in the file type dropdown
   • Once loaded, click “Save private key”
   • Save the file with .ppk extension

6. Test New User Access

   • Keep your original SSH session open
   • Open a new PuTTY session
   • Configure PuTTY with:
     • Host: Your server’s address
     • Username: secureuser
     • Auth: Select your new .ppk file
   • Test the connection

7. Safety Checks

   • Verify you can log in with the new user
   • Confirm sudo privileges if required
   • Only after successful testing, close the original session

Important Safety Notes:

• Store your key files securely
• Make backup copies of both .key and .ppk files

[hhf.technoloy]

@player684

Would you recommend Falco or Crowdsec or both? What is the best setup in your opinion?

Crowdsec preferred. Falco is over kill

[player684]

Great. You have give me a lot to work with today

I am gonna follow your crowdsec guide and see if I can figure everything and the above to see if everything will work as I want.

Thank you so much.

[hhf.technoloy]

@hhf.technoloy

Note: Keep the terminal window open during the entire process. donot not close it

@player684 make sure tailscale client is running on your pc and approve the both on the tailscale dashboard-- vm and the pc.

[player684]

BTW I hope more people find your forum, you have created some really useful resources. Thanks for all the effort

[hhf.technoloy]

its little complicated. you can DM me on discord if you get stuck

[hhf.technoloy]

Thank you. its to benefit all the community members. I hope people utilize more and more.

[player684]

so I added the monitoring service to discord but I am getting auth failed.

[player684]

Is this not because i have 2FA on Panoglin?

edit: I created a new admin user in Pangolin without 2FA and now stopped getting auth error in Discord

[player684]

Hi, I implemented the monitoring service only with the script. It was working brilliantly, but today, for some reason, it stated sending all kinds of errors. I tried to troubleshoot but not getting anywhere. Is there anything I can do to get this back up?

I have now stopped the systemd service, but this is what it looked like on Discord

Screenshot-20250207-111012 hosted at ImgBB

[hhf.technoloy]

share your logs and your pangolin version details

[player684]

Is it okay to share publicly here? or Can you please send me a DM and I will reply wih logs.

[german]

hey i edited this version and added pushover https://raw.githubusercontent.com/synologyy/pango-scripts/refs/heads/main/pangolin-monitor.sh

[hhf.technoloy]

Its awesome. i just tested it out. work perfectly.