Enhanced Unraid ZFS Error Notification Script with Discord Integration or without

Servers Home Servers
, , ,
1

Let me explain the how the script works:

  1. Script Structure and Error Handling:

    • Added set -euo pipefail for strict error handling
    • Organized code into functions for better modularity and maintenance
    • Added proper logging functionality
    • Added ZFS availability check

  2. Key Functions:

    • log_message: Records script activity with timestamps
    • check_zfs_available: Verifies ZFS commands are available
    • get_pool_status: Safely gets pool status with error handling
    • check_pool_health: Determines severity based on pool state
    • send_notification: Handles notification logic with duplicate checking
    • main: Orchestrates the overall script flow

  3. Core Functionality:

    • Checks all ZFS pools for errors
    • Monitors for DEGRADED or FAULTED states
    • Sends notifications only for mounted pools with issues
    • Prevents duplicate notifications using checksums
    • Supports both warning and alert severity levels

  4. Enhanced Features:

    • Better error handling and logging
    • More robust pool status checking
    • Clearer variable names and documentation
    • Array handling improvements using mapfile
    • Configuration variables at the top for easy modification

  5. How It Works:

    1. Script starts and checks for ZFS availability
2. Gets list of all ZFS pools
3. Checks overall status of all pools
4. For each pool:
   - Verifies if it's mounted
   - Checks individual pool status
   - Records errors and determines severity
5. If errors are found:
   - Generates checksum to prevent duplicates
   - Sends notification with appropriate severity
   - Logs the action

The script can be used in two ways:

  1. Manual Execution:

    chmod +x zfs_notify.sh
./zfs_notify.sh

  2. Automated via Cron:

    # Add to crontab (run every hour)
0 * * * * /path/to/zfs_notify.sh

To monitor the script’s activity, you can check the log file:

tail -f /var/log/zfs_errors.log

[zfs_notify.sh]

#!/bin/bash

# Set strict error handling
set -euo pipefail

# Configuration
NOTIFY_SCRIPT="/usr/local/emhttp/webGui/scripts/notify"
LOG_FILE="/var/log/zfs_errors.log"

# Function to log messages
log_message() {
    local message="$1"
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    echo "[$timestamp] $message" >> "$LOG_FILE"
}

# Function to check if ZFS is available
check_zfs_available() {
    if ! command -v zfs >/dev/null 2>&1; then
        log_message "ERROR: ZFS commands not found. Is ZFS installed?"
        exit 1
    }
}

# Function to get pool status
get_pool_status() {
    local pool="$1"
    zpool status -x "$pool" 2>/dev/null || echo "Error getting status for pool $pool"
}

# Function to check pool health
check_pool_health() {
    local status="$1"
    local severity="warning"
    
    if echo "$status" | grep -q "DEGRADED"; then
        severity="alert"
    elif echo "$status" | grep -q "FAULTED"; then
        severity="alert"
    fi
    
    echo "$severity"
}

# Function to send notification
send_notification() {
    local severity="$1"
    local pools="$2"
    local status="$3"
    local cksum="$4"
    
    # Check for existing notifications
    if /usr/local/emhttp/webGui/scripts/notify get | grep -q "$cksum"; then
        log_message "Notification already exists for checksum: $cksum"
        return 0
    }
    
    # Send notification
    "$NOTIFY_SCRIPT" \
        -i "$severity" \
        -e "ZFS" \
        -s "ZFS Errors Detected" \
        -d "Errors detected in the following pools:<br>${pools}<br>($cksum)" \
        -m "$status"
        
    log_message "Sent notification for pools: $pools"
}

main() {
    # Check if ZFS is available
    check_zfs_available
    
    # Get all ZFS pools
    mapfile -t pools < <(zfs list -d 0 -o name -H)
    
    # Get overall status
    zStatus=$(zpool status -x -v)
    
    # Initialize arrays and variables
    declare -a poolErr=()
    severity="warning"
    
    # Check if there are any errors
    if [ "$(echo "$zStatus" | wc -l)" -gt 1 ]; then
        # Loop through each pool
        for pool in "${pools[@]}"; do
            # Check if pool is mounted
            if mountpoint -q "/mnt/$pool"; then
                pool_status=$(get_pool_status "$pool")
                
                # Check for errors in this pool
                if [ "$(echo "$pool_status" | wc -l)" -gt 1 ]; then
                    poolErr+=("$pool")
                    
                    # Update severity if needed
                    current_severity=$(check_pool_health "$pool_status")
                    if [ "$current_severity" = "alert" ]; then
                        severity="alert"
                    fi
                fi
            fi
        done
    fi
    
    # Send notification if there are errors
    if [ "${#poolErr[@]}" -gt 0 ]; then
        cksum=$(echo "$zStatus" | cksum | cut -f1 -d' ')
        send_notification "$severity" "${poolErr[*]}" "$zStatus" "$cksum"
    else
        log_message "No ZFS errors detected"
    fi
}

# Run main function
main "$@"

Enhanced Unraid ZFS Error Notification Script with Discord Integration

I have modify the script to include Discord webhook notifications alongside the existing Unraid notifications. This will allow for monitoring ZFS errors through Discord as well.

I’ve enhanced the script to include Discord notifications. Here are the key additions and how to set it up:

  1. New Discord Configuration Section:
# Discord Configuration
DISCORD_WEBHOOK_URL="YOUR_DISCORD_WEBHOOK_URL"
DISCORD_USERNAME="Unraid ZFS Monitor"
DISCORD_AVATAR_URL="https://i.imgur.com/your-avatar.png"
  1. New Discord Notification Function:
  • Creates formatted messages with embeds
  • Color-coded based on severity (red for alerts, yellow for warnings)
  • Includes timestamp and pool details
  • Formats ZFS status output in a code block
  • Handles message length limits
  • Includes error handling for failed notifications
  1. Setting Up Discord Integration:

a. Create a Discord Webhook:

  1. Open Discord server settings
  2. Go to “Integrations” → “Webhooks”
  3. Click “New Webhook”
  4. Name it (e.g., “Unraid ZFS Monitor”)
  5. Choose the channel for notifications
  6. Copy the webhook URL

b. Configure the Script:

# Edit the script and replace the webhook URL
DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/your-webhook-url"
  1. Sample Discord Notification Format:
Title: ZFS Errors Detected
Description: Issues found in the following pools:
pool1, pool2

Severity: alert

Details:
[ZFS status output in code block]

Footer: Unraid ZFS Monitor - [Timestamp]
  1. Using the Script:

a. Make it executable:

chmod +x zfs_notify.sh

b. Test the Discord integration:

./zfs_notify.sh

c. Set up automated monitoring via cron:

# Edit crontab
crontab -e

# Add this line to run every hour
0 * * * * /path/to/zfs_notify.sh
  1. Features of Discord Integration:
  • Color-coded notifications based on severity
  • Formatted message embeds for better readability
  • Code blocks for status output
  • Timestamp information
  • Error handling and logging
  • Configurable bot name and avatar
  1. Troubleshooting:
  • Check the log file for notification status:
tail -f /var/log/zfs_errors.log
  • Test the Discord webhook separately:
curl -H "Content-Type: application/json" \
     -d '{"content": "Test message"}' \
     YOUR_DISCORD_WEBHOOK_URL

#!/bin/bash

# Set strict error handling
set -euo pipefail

# Configuration
NOTIFY_SCRIPT="/usr/local/emhttp/webGui/scripts/notify"
LOG_FILE="/var/log/zfs_errors.log"

# Discord Configuration - Replace with your webhook URL
DISCORD_WEBHOOK_URL="YOUR_DISCORD_WEBHOOK_URL"
DISCORD_USERNAME="Unraid ZFS Monitor"
DISCORD_AVATAR_URL="https://i.imgur.com/your-avatar.png"  # Optional: Replace with your desired avatar URL

# Function to log messages
log_message() {
    local message="$1"
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    echo "[$timestamp] $message" >> "$LOG_FILE"
}

# Function to check if ZFS is available
check_zfs_available() {
    if ! command -v zfs >/dev/null 2>&1; then
        log_message "ERROR: ZFS commands not found. Is ZFS installed?"
        exit 1
    }
}

# Function to send Discord notification
send_discord_notification() {
    local severity="$1"
    local pools="$2"
    local status="$3"
    
    # Set color based on severity (Discord uses decimal color codes)
    local color
    if [ "$severity" = "alert" ]; then
        color="16711680"  # Red
    else
        color="16776960"  # Yellow
    fi
    
    # Format status message for Discord (limit to 2000 characters)
    local formatted_status
    formatted_status=$(echo "$status" | sed 's/```/\\`\\`\\`/g' | cut -c 1-1500)
    
    # Create JSON payload
    local json_payload=$(cat <<EOF
{
  "username": "${DISCORD_USERNAME}",
  "avatar_url": "${DISCORD_AVATAR_URL}",
  "embeds": [
    {
      "title": "ZFS Errors Detected",
      "description": "Issues found in the following pools:\n${pools}",
      "color": ${color},
      "fields": [
        {
          "name": "Severity",
          "value": "${severity}",
          "inline": true
        },
        {
          "name": "Details",
          "value": "\`\`\`${formatted_status}\`\`\`"
        }
      ],
      "footer": {
        "text": "Unraid ZFS Monitor - $(date '+%Y-%m-%d %H:%M:%S')"
      }
    }
  ]
}
EOF
)
    
    # Send to Discord
    curl -H "Content-Type: application/json" \
         -d "$json_payload" \
         "$DISCORD_WEBHOOK_URL" 2>/dev/null || {
        log_message "ERROR: Failed to send Discord notification"
        return 1
    }
    
    log_message "Sent Discord notification for pools: $pools"
}

# Function to get pool status
get_pool_status() {
    local pool="$1"
    zpool status -x "$pool" 2>/dev/null || echo "Error getting status for pool $pool"
}

# Function to check pool health
check_pool_health() {
    local status="$1"
    local severity="warning"
    
    if echo "$status" | grep -q "DEGRADED"; then
        severity="alert"
    elif echo "$status" | grep -q "FAULTED"; then
        severity="alert"
    fi
    
    echo "$severity"
}

# Function to send Unraid notification
send_unraid_notification() {
    local severity="$1"
    local pools="$2"
    local status="$3"
    local cksum="$4"
    
    # Check for existing notifications
    if /usr/local/emhttp/webGui/scripts/notify get | grep -q "$cksum"; then
        log_message "Notification already exists for checksum: $cksum"
        return 0
    }
    
    # Send notification
    "$NOTIFY_SCRIPT" \
        -i "$severity" \
        -e "ZFS" \
        -s "ZFS Errors Detected" \
        -d "Errors detected in the following pools:<br>${pools}<br>($cksum)" \
        -m "$status"
        
    log_message "Sent Unraid notification for pools: $pools"
}

main() {
    # Check if ZFS is available
    check_zfs_available
    
    # Get all ZFS pools
    mapfile -t pools < <(zfs list -d 0 -o name -H)
    
    # Get overall status
    zStatus=$(zpool status -x -v)
    
    # Initialize arrays and variables
    declare -a poolErr=()
    severity="warning"
    
    # Check if there are any errors
    if [ "$(echo "$zStatus" | wc -l)" -gt 1 ]; then
        # Loop through each pool
        for pool in "${pools[@]}"; do
            # Check if pool is mounted
            if mountpoint -q "/mnt/$pool"; then
                pool_status=$(get_pool_status "$pool")
                
                # Check for errors in this pool
                if [ "$(echo "$pool_status" | wc -l)" -gt 1 ]; then
                    poolErr+=("$pool")
                    
                    # Update severity if needed
                    current_severity=$(check_pool_health "$pool_status")
                    if [ "$current_severity" = "alert" ]; then
                        severity="alert"
                    fi
                fi
            fi
        done
    fi
    
    # Send notifications if there are errors
    if [ "${#poolErr[@]}" -gt 0 ]; then
        cksum=$(echo "$zStatus" | cksum | cut -f1 -d' ')
        
        # Send both Unraid and Discord notifications
        send_unraid_notification "$severity" "${poolErr[*]}" "$zStatus" "$cksum"
        send_discord_notification "$severity" "${poolErr[*]}" "$zStatus"
    else
        log_message "No ZFS errors detected"
    fi
}

# Run main function
main "$@"