Guide Version: 1.0
Last Updated: Friday, May 16, 2025
Introduction:
This guide will walk you through integrating VictoriaMetrics with your existing CrowdSec instance (as part of a “Pangolin-like” Docker setup) and visualizing rich threat intelligence data in Grafana. We’ll leverage CrowdSec’s HTTP notifications to send detailed alert data to VictoriaMetrics, which Grafana will then use as a data source. This provides a powerful, self-hosted alternative to CrowdSec’s default metrics, offering deeper insights into the threats your services are facing, including geolocation, ASN information, and Traefik router details.
This guide assumes you’re using a setup similar to the “Pangolin” configuration, which typically involves:
- A main
docker-compose.ymlfor services like Traefik, CrowdSec, and other applications. - A separate
docker-compose-monitoring.ymlfor Prometheus, Grafana, etc. - Both compose setups are linked via a shared external Docker network (e.g.,
pangolin). - CrowdSec is configured to monitor Traefik logs.
Prerequisites:
- A working “Pangolin-like” Docker environment with:
- CrowdSec running and configured (especially for Traefik logs).
- Grafana running.
- Both CrowdSec and Grafana (and soon VictoriaMetrics) connected to the same shared Docker network (referred to as
pangolinin this guide). - Your main services defined in a
docker-compose.yml(e.g., located at~/Pangolin/docker-compose.yml). - Your monitoring services defined in
docker-compose-monitoring.yml(e.g., at~/Pangolin/docker-compose-monitoring.yml). - A directory structure like
~/Pangolin/config/crowdsec/for CrowdSec configurations.
dockeranddocker-composeinstalled on your host.- Basic familiarity with editing YAML files and running Docker commands.
Overview of Changes:
- Add VictoriaMetrics as a new service in your
docker-compose-monitoring.yml. - Create a new CrowdSec HTTP notification handler to format and send alert data.
- Update your CrowdSec profiles to use this new notification handler.
- Configure Grafana to use VictoriaMetrics as a data source.
- Import a pre-built Grafana dashboard (ID
21689) to visualize the data.
Step-by-Step Setup Process:
Step 1: Prepare Directory for VictoriaMetrics Data
VictoriaMetrics needs a persistent volume to store its time-series data. We’ll create a directory for this within your existing configuration structure.
On your Docker host, run:
mkdir -p ~/Pangolin/config/victoriametrics_data
Step 2: Add VictoriaMetrics to Your Monitoring Docker Compose File
Edit your docker-compose-monitoring.yml file (e.g., ~/Pangolin/docker-compose-monitoring.yml) and add the following service definition for victoriametrics. Place it alongside your existing prometheus, grafana services:
networks:
pangolin: # Ensure this matches your existing external network name
external: true
services:
# ... your existing prometheus, grafana, node-exporter, etc. services ...
victoriametrics: # <-- NEW SERVICE DEFINITION STARTS HERE
image: victoriametrics/victoria-metrics:latest
container_name: victoriametrics
restart: unless-stopped
expose: # Exposes port 8428 only to other containers on the 'pangolin' network
- "8428"
# If you need to test VictoriaMetrics directly from your host (e.g., with curl),
# you can temporarily change 'expose:' to 'ports: ["8428:8428"]'
volumes:
- ./config/victoriametrics_data:/victoria-metrics-data # Maps to ~/Pangolin/config/victoriametrics_data
command:
- '--storageDataPath=/victoria-metrics-data'
- '--httpListenAddr=:8428' # Listen on all interfaces within the container on port 8428
- '--retentionPeriod=3' # Example: Retain data for 3 months. Adjust as needed (e.g., "12m" for 12 months).
networks:
- pangolin # Connect to your shared network
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
# <-- NEW SERVICE DEFINITION ENDS HERE
# prometheus:
# ... (rest of your prometheus config) ...
# grafana:
# ... (rest of your grafana config) ...
expose: - "8428": This makes VictoriaMetrics available to other containers on thepangolinnetwork (like CrowdSec and Grafana) viahttp://victoriametrics:8428.volumes: The path./config/victoriametrics_datais relative to the location of yourdocker-compose-monitoring.yml.retentionPeriod: Adjust this based on how long you want to keep the detailed alert data.
Step 3: Create the CrowdSec HTTP Notification Handler
This file tells CrowdSec how to format and where to send the alert data.
Create a new file named http_victoriametrics.yaml in your CrowdSec notifications directory on the host:
Path: ~/Pangolin/config/crowdsec/notifications/http_victoriametrics.yaml
type: http
name: http_victoriametrics
log_level: info # Change to 'debug' for more verbose logging during troubleshooting
format: >
{{- range $Alert := . -}}
{{- $traefikRouters := GetMeta . "traefik_router_name" -}}
{{- $instanceName := env "ENROLL_INSTANCE_NAME" -}}
{{- if not $instanceName -}}
{{- $instanceName = "pangolin-crowdsec-default" -}}
{{- end -}}
{{- range .Decisions -}}
{"metric":{"__name__":"cs_lapi_decision","instance":"{{$instanceName}}","country":"{{$Alert.Source.Cn}}","asname":"{{$Alert.Source.AsName}}","asnumber":"{{$Alert.Source.AsNumber}}","latitude":"{{$Alert.Source.Latitude}}","longitude":"{{$Alert.Source.Longitude}}","iprange":"{{$Alert.Source.Range}}","scenario":"{{.Scenario}}","type":"{{.Type}}","duration":"{{.Duration}}","scope":"{{.Scope}}","ip":"{{.Value}}","traefik_routers":{{ printf "%q" ($traefikRouters | uniq | join ",")}}},"values": [1],"timestamps":[{{now.UTC.UnixEpoch}}000]}
{{- end }}
{{- end -}}
url: http://victoriametrics:8428/api/v1/import
method: POST
headers:
Content-Type: "application/json"
#Authorization: "<SECRET-TOKEN>" # Uncomment and configure if you set up vmauth for VictoriaMetrics later
instance: This uses theENROLL_INSTANCE_NAMEenvironment variable from your CrowdSec Docker Compose setup, ensuring a meaningful instance tag.traefik_routers: This extracts the Traefik router name if available in the alert metadata (useful if CrowdSec is parsing Traefik logs).url: Points to the VictoriaMetrics service on your shared Docker network.
Step 4: Update CrowdSec Profiles
Now, tell CrowdSec to use this new notification handler. Edit your existing CrowdSec profiles file:
Path: ~/Pangolin/config/crowdsec/profiles.yaml
Add http_victoriametrics to the notifications: list within the profiles whose alerts you want to send to VictoriaMetrics. For example:
# ... (other profiles like captcha_remediation if you have them) ...
# ---
name: default_ip_remediation # Example profile
filters:
- Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
- type: ban
duration: 4h
notifications:
- gotify # Your existing notification
- discord # Your existing notification
- http_victoriametrics # <-- ADD THIS LINE
on_success: break
---
name: default_range_remediation # Example profile
filters:
- Alert.Remediation == true && Alert.GetScope() == "Range"
decisions:
- type: ban
duration: 4h
notifications:
- gotify # Your existing notification
- discord # Your existing notification
- http_victoriametrics # <-- ADD THIS LINE
on_success: break
# Add to other profiles as needed
- Review all profiles in your
profiles.yamland addhttp_victoriametricswhere appropriate.
Step 5: Restart Services
Apply the changes:
-
Restart CrowdSec: Navigate to the directory containing your main
docker-compose.yml(e.g.,~/Pangolin/) and run:cd ~/Pangolin docker-compose restart crowdsecCheck CrowdSec logs for errors:
docker-compose logs -f crowdsec -
Start/Update Monitoring Stack (including VictoriaMetrics): Navigate to the directory containing
docker-compose-monitoring.yml(e.g.,~/Pangolin/) and run:cd ~/Pangolin docker-compose -f docker-compose-monitoring.yml up -d --remove-orphansCheck VictoriaMetrics logs:
docker-compose -f docker-compose-monitoring.yml logs -f victoriametrics
Step 6: Verify CrowdSec GeoIP Parser
For map visualizations and country data, ensure CrowdSec’s GeoIP enrichment is active:
# Assuming your CrowdSec container is named 'crowdsec'
docker exec -it crowdsec cscli parsers install crowdsecurity/geoip-enrich
docker exec -it crowdsec cscli collections install crowdsecurity/geoip # Recommended for modern GeoIP handling
docker exec -it crowdsec cscli hub update
docker exec -it crowdsec cscli hub upgrade
After running these, restart CrowdSec again:
cd ~/Pangolin
docker-compose restart crowdsec
Step 7: Configure Grafana
-
Add VictoriaMetrics as a Data Source:
- Open your Grafana instance in a web browser.
- Navigate to Connections (plug icon in the side menu) -> Data Sources.
- Click “Add new data source”.
- Search for and select “Prometheus” (VictoriaMetrics is Prometheus-compatible).
- Name: Enter a descriptive name, e.g.,
VictoriaMetrics-CrowdSec. - URL:
http://victoriametrics:8428 - Leave other settings as default for now.
- Click “Save & Test”. You should see a success message (“Data source is working”).
-
Import the Grafana Dashboard:
- Navigate to Dashboards (four squares icon in the side menu).
- Click “New” (top right button) -> “Import”.
- In the “Import via grafana.com” field, enter the Dashboard ID:
21689. - Click “Load”.
- On the next page, you can customize the dashboard name if desired.
- Under “Data Sources” (or similar options at the bottom), ensure you select your newly created
VictoriaMetrics-CrowdSecdata source from the dropdown list(s). - Click “Import”.
Step 8: Verify and Test
- Generate Test Alerts: If you don’t have recent CrowdSec activity, generate a test decision:
docker exec -it crowdsec cscli decisions add --ip 1.2.3.4 --reason test-dashboard --duration 10m - Check CrowdSec Logs:
Look for logs from thecd ~/Pangolin docker-compose logs --tail 50 -f crowdsechttp_victoriametricsplugin indicating it’s sending data (especially iflog_levelisdebugin the notification template). There should be no template errors. - Query VictoriaMetrics Directly (Optional):
You can quickly check if data is arriving in VictoriaMetrics from another container on thepangolinnetwork (e.g., Grafana, if it hascurl):
You should see a JSON response. Ifdocker exec -it grafana curl "http://victoriametrics:8428/api/v1/query?query=cs_lapi_decision"resultis not empty, data is there. - View the Grafana Dashboard: Open the imported “CrowdSec Cyber Threat Insights” dashboard in Grafana. Adjust the time range (e.g., “Last 15 minutes”) to see recent events. Data should start populating the panels.
Troubleshooting Common Issues:
- No data in Grafana:
- Verify CrowdSec is generating alerts/decisions.
- Check CrowdSec logs for any errors from the
http_victoriametricsplugin. Theenvtemplate error we troubleshooted earlier is a common pitfall if the template is incorrect. - Ensure VictoriaMetrics is running and accessible (
docker exec -it grafana ping victoriametrics). - Confirm the correct data source is selected in the Grafana dashboard panels/variables.
- Check the Grafana time range.
- Map not showing data: Ensure the GeoIP steps were completed and CrowdSec restarted.
- Template Errors in CrowdSec logs: Double-check the syntax of your
http_victoriametrics.yamlfile against the example provided, especially the Go template section.
Conclusion:
You should now have a robust Cyber Threat Intelligence dashboard powered by your Pangolin CrowdSec instance, VictoriaMetrics, and Grafana! This setup provides valuable, self-hosted insights into the security events affecting your services. Feel free to customize the Grafana dashboard further to suit your specific needs.
Happy monitoring!