Detailed Guide: Creating a Grafana Dashboard for Traefik Metrics in Pangolin Stack- Part 2

[hhf.technoloy]

Detailed Guide: Creating a Grafana Dashboard for Traefik Metrics in Pangolin Stack

I’ll guide you through setting up a Grafana dashboard to monitor your Traefik metrics in the Pangolin stack step by step.

Part 1: Enable Metrics in Traefik

Step 1: Edit Traefik Configuration

1. Edit your Traefik configuration file:

nano config/traefik/traefik_config.yml

2. Add the Prometheus metrics endpoint by adding this section:

metrics:
  prometheus:
    addEntryPointsLabels: true
    addServicesLabels: true
    addRoutersLabels: true
    buckets:
      - 0.1
      - 0.3
      - 1.2
      - 5.0

3. Your updated file should include this section (make sure it’s at the same indentation level as other main sections like api:, providers:, etc.):

################################################################
# API and Dashboard
################################################################

api:
  insecure: true
  dashboard: true

################################################################
# Metrics
################################################################

metrics:
  prometheus:
    addEntryPointsLabels: true
    addServicesLabels: true
    addRoutersLabels: true
    buckets:
      - 0.1
      - 0.3
      - 1.2
      - 5.0

################################################################
# Providers
################################################################

Part 2: Setting Up Prometheus

Step 1: Create a Prometheus Configuration File

1. Create a directory for Prometheus:

mkdir -p config/prometheus

2. Create a prometheus.yml file:

nano config/prometheus/prometheus.yml

3. Add the following configuration:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'traefik'
    static_configs:
      - targets: ['gerbil:8080']
    metrics_path: /metrics
  
  - job_name: 'crowdsec'
    static_configs:
      - targets: ['crowdsec:6060']

Step 2: Add Prometheus to Your Docker Compose

1. Edit your docker-compose.yml file:

nano docker-compose.yml

2. Add the Prometheus service:

prometheus:
  image: prom/prometheus:latest
  container_name: prometheus
  networks:
    - pangolin
  volumes:
    - ./config/prometheus:/etc/prometheus
    - prometheus-data:/prometheus
  command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--storage.tsdb.path=/prometheus'
    - '--web.console.libraries=/etc/prometheus/console_libraries'
    - '--web.console.templates=/etc/prometheus/consoles'
  ports:
    - "9091:9090"
  restart: unless-stopped

3. Add a volumes section at the end of your docker-compose file if it doesn’t exist:

volumes:
  prometheus-data:

Step 3: Modify Traefik Service to Expose Metrics

1. In your docker-compose.yml, make sure your Traefik service exposes port 8080:

traefik:
  image: traefik:v3.3.3
  container_name: traefik
  restart: unless-stopped
  network_mode: service:gerbil
  depends_on:
    pangolin:
      condition: service_healthy
  command:
    - --configFile=/etc/traefik/traefik_config.yml
    # Add metrics endpoint
    - --metrics.prometheus=true
    - --metrics.prometheus.addServicesLabels=true
    - --metrics.prometheus.addEntryPointsLabels=true
  volumes:
    - ./config/traefik:/etc/traefik:ro
    - ./config/letsencrypt:/letsencrypt
    - ./config/traefik/logs:/var/log/traefik
    - ./config/traefik/conf/:/etc/traefik/conf/
    - ./config/traefik/rules:/rules

2. In the gerbil service, make sure port 8080 is exposed:

gerbil:
  # your existing configuration
  ports:
    - 51820:51820/udp
    - 443:443
    - 80:80
    - 8085:8080  # Add this for Traefik metrics

Part 3: Setting Up Grafana

Step 1: Add Grafana to Docker Compose

1. Add the Grafana service to your docker-compose.yml:

grafana:
  image: grafana/grafana:latest
  container_name: grafana
  networks:
    - pangolin
  volumes:
    - grafana-data:/var/lib/grafana
  environment:
    - GF_SECURITY_ADMIN_USER=admin
    - GF_SECURITY_ADMIN_PASSWORD=strongpassword
    - GF_USERS_ALLOW_SIGN_UP=false
  ports:
    - "3035:3000"
  restart: unless-stopped

2. Add to the volumes section:

volumes:
  prometheus-data:
  grafana-data:

Step 2: Apply Changes

1. Apply the changes to your Docker Compose stack:

docker compose down
docker compose up -d

2. Verify all services are running:

docker ps

Step 3: Configure Grafana

1. Access Grafana in your browser:

   • http://your-server-ip:3035

2. Log in with:

   • Username: admin
   • Password: strongpassword (or whatever you set in docker-compose)

3. Add Prometheus as a data source:

   • Click on “Configuration” (gear icon) in the left sidebar
   • Select “Data sources”
   • Click “Add data source”
   • Select “Prometheus”
   • Set URL to: http://prometheus:9090
   • Click “Save & Test” to verify connection

Part 4: Creating the Traefik Dashboard

Method 1: Import a Pre-made Dashboard

1. In Grafana, click the “+” icon in the left sidebar
2. Select “Import”
3. Enter one of these dashboard IDs:
   • 17346 (Traefik Dashboard)
   • 2870 (Traefik v3 Dashboard)
4. Click “Load”
5. Select your Prometheus data source
6. Click “Import”

Method 2: Build a Custom Dashboard

If you prefer to build your own dashboard, follow these steps:

1. Click the “+” icon in the left sidebar

2. Select “Create Dashboard”

3. Click “Add new panel”

4. Create a panel for Request Rate:

   • In the Query field enter: sum(rate(traefik_service_requests_total[1m])) by (service)
   • Panel title: “Requests per minute by service”
   • Visualization: “Time series”
   • Click “Apply”

5. Create a panel for Response Status Codes:

   • Click “Add panel”
   • Query: sum(traefik_service_requests_total) by (code)
   • Panel title: “Response Status Codes”
   • Visualization: “Pie Chart”
   • Click “Apply”

6. Create a panel for Request Duration:

   • Click “Add panel”
   • Query: histogram_quantile(0.95, sum(rate(traefik_service_request_duration_seconds_bucket[5m])) by (le, service))
   • Panel title: “Request Duration (95th percentile)”
   • Visualization: “Time series”
   • Click “Apply”

7. Create a panel for Connections:

   • Click “Add panel”
   • Query: traefik_entrypoint_open_connections{entrypoint="websecure"}
   • Panel title: “Open Connections (HTTPS)”
   • Visualization: “Stat”
   • Click “Apply”

8. Save your dashboard:

   • Click the save icon in the top right
   • Name: “Traefik Monitoring”
   • Click “Save”

Part 5: Advanced Traefik Metrics

Here are some additional panels you might want to add:

1. Add a panel for Error Rates:

   • New panel
   • Query: sum(rate(traefik_service_requests_total{code=~"5.*"}[1m])) by (service) / sum(rate(traefik_service_requests_total[1m])) by (service)
   • Title: “Error Rate by Service”
   • Visualization: “Time series”

2. Add a panel for Entrypoint Traffic:

   • New panel
   • Query: sum(rate(traefik_entrypoint_requests_total[1m])) by (entrypoint)
   • Title: “Requests by Entrypoint”
   • Visualization: “Time series”

3. Add a panel for Top Routes:

   • New panel
   • Query: topk(10, sum(traefik_router_requests_total) by (router))
   • Title: “Top 10 Routes”
   • Visualization: “Bar gauge”

Part 6: Troubleshooting

If you don’t see metrics:

1. Check if Traefik is exposing metrics:

curl http://localhost:8080/metrics

2. Verify Prometheus can reach Traefik:

docker exec -it prometheus wget -O- gerbil:8080/metrics

3. Check Prometheus logs:

docker logs prometheus

4. Check if Traefik metrics are in Prometheus:

   • Go to http://your-server-ip:9090
   • Try querying: traefik_entrypoint_requests_total

5. Make sure ports are correctly exposed:

docker-compose ps

Part 7: Setting Up Alerts (Optional)

1. In Grafana, go to Alerting in the left sidebar
2. Click “New alert rule”
3. Configure a rule, for example:
   • Query: sum(rate(traefik_service_requests_total{code=~"5.*"}[1m])) > 5
   • Condition: When value is above 5
   • For: 5m
   • Set rule name: “High error rate”
4. Set notification channels
5. Save the rule

That’s it! You now have a complete Grafana dashboard to monitor your Traefik metrics from your Pangolin stack.

[nwilliamsscu]

How does this work with the new network_mode / default bridge that they’ve got in the v1 docker-compose.yml? Ultimately, it looks like prometheus doesn’t have access to traefik via docker network, even though they all seem to share the default bridge “pangolin” network?

networks:
  default:
    driver: bridge
    name: pangolin

    image: traefik:v3.3.3
    network_mode: service:gerbil

[nwilliamsscu]

@hhf.technoloy

docker exec -it prometheus wget -O- traefik:8080/metrics

it looks like this should be docker exec -it prometheus wget -O- gerbil:8080/metrics with the new network_mode: service:gerbil setting…

… as such, I modified the config/prometheus/prometheus.yml file as follows:

scrape_configs:
  - job_name: 'traefik'
    static_configs:
      - targets: ['gerbil:8080']
    metrics_path: /metrics