Part 3: Testing and Advanced Configuration of CrowdSec with Pangolin
Use with your own discretion. version 1.6.6 has auto allowlist command also please refer crowdsec documentation.
1. Testing CrowdSec Integration
1.1 Initial Testing Setup
# Verify CrowdSec is processing logs
docker exec -it crowdsec cscli metrics
# Check current decisions
docker exec -it crowdsec cscli decisions list
# Monitor live logs
docker exec -it crowdsec tail -f /var/log/crowdsec.log
1.2 Testing with Sample Data
Test different scenarios:
# Test ban scenario
docker exec crowdsec cscli decisions add -i 192.168.1.1 -t ban -d 1h
# Test captcha scenario
docker exec crowdsec cscli decisions add --ip 192.168.1.2 --type captcha -d 1h
# Verify decisions were added
docker exec -it crowdsec cscli decisions list
# Test removing decisions
docker exec crowdsec cscli decisions delete -i 192.168.1.1
1.3 Log Analysis
# Analyze Traefik logs
docker exec -it crowdsec cscli explain /var/log/traefik/traefik.log
# Check AppSec metrics
curl http://localhost:6060/metrics | grep appsec
# Monitor Traefik logs
docker compose logs traefik -f
2. Advanced Configuration
2.1 Custom Scenarios
Create /config/crowdsec/scenarios/custom-scenarios.yaml:
type: trigger
name: custom-ban-bruteforce
description: "Ban IPs attempting brute force"
filter: "evt.Meta.log_type == 'traefik_access'"
groupBy: evt.Meta.source_ip
distinctBy: evt.Meta.http_path
capacity: 5
leakSpeed: "10s"
blackhole: 1m
labels:
service: traefik
type: bruteforce
remediation: true
2.2 Custom Parsers
Create /config/crowdsec/parsers/custom-parser.yaml:
name: custom-traefik-parser
description: "Custom parser for Traefik logs"
filter: "evt.Meta.log_type == 'traefik_access'"
onsuccess: next_stage
pattern_syntax: "LTSV"
patterns:
- CUSTOM_TRAEFIK_LOG time=%{TIMESTAMP_ISO8601:timestamp}\tlevel=%{WORD:level}\tmsg=%{GREEDYDATA:message}
nodes:
- grok:
pattern: CUSTOM_TRAEFIK_LOG
statics:
- meta: log_type
value: traefik_access
- meta: service
value: traefik
2.3 Custom Collections
Create /config/crowdsec/collections/custom-collection.yaml:
parsers:
- custom-traefik-parser
scenarios:
- custom-ban-bruteforce
descriptions:
custom-ban-bruteforce: "Ban IPs attempting brute force attacks"
3. Monitoring and Metrics
3.1 Prometheus Metrics
Update /config/traefik/traefik_config.yml:
metrics:
prometheus:
entryPoint: metrics
addServicesLabels: true
addEntryPointsLabels: true
buckets:
- 0.1
- 0.3
- 1.2
- 5.0
3.2 Dashboard Integration
Enable metrics in CrowdSec configuration:
prometheus:
enabled: true
level: full
listen_addr: 0.0.0.0
listen_port: 6060
4. Advanced Security Configurations
4.1 Rate Limiting
Update Traefik dynamic config:
http:
middlewares:
rate-limit:
rateLimit:
average: 100
burst: 50
period: 1m
custom-security:
chain:
middlewares:
- rate-limit
- crowdsec
4.2 IP Whitelisting
Update CrowdSec configuration:
whitelist:
ips:
- "10.0.0.0/8"
- "172.16.0.0/12"
- "192.168.0.0/16"
cidrs:
- "100.89.137.0/20"
5. Advanced Troubleshooting
5.1 Debug Mode
Enable debug logging:
# Set CrowdSec to debug mode
docker exec -it crowdsec cscli config show --debug
# Enable debug logging in Traefik
sed -i 's/level: "INFO"/level: "DEBUG"/' /config/traefik/traefik_config.yml
5.2 Testing Tools
# Test bouncer connectivity
curl -X GET http://localhost:9090/v1/decisions -H "X-Api-Key: <your_bouncer_key>"
# Test AppSec WAF
curl -X POST http://localhost:7422/verify -H "Content-Type: application/json" -d '{"request":{"uri":"/test"}}'
# Verify log parsing
docker exec -it crowdsec cscli parsers debug --file /var/log/traefik/access.log
5.3 Common Issues and Solutions
- Bouncer Authentication Issues:
# Regenerate bouncer key
docker exec crowdsec cscli bouncers delete traefik-bouncer
docker exec crowdsec cscli bouncers add traefik-bouncer
- Log Processing Issues:
# Check acquisition status
docker exec crowdsec cscli metrics
- Network Connectivity:
# Test internal networking
docker exec -it crowdsec ping traefik
docker exec -it crowdsec ping pangolin
6. Performance Tuning
6.1 CrowdSec Optimization
Update /config/crowdsec/config.yaml:
crowdsec_service:
parser_routines: 4
acquisition_path: /etc/crowdsec/acquis.yaml
acquisition_dir: /etc/crowdsec/acquis.d
db_config:
flush:
max_items: 5000
max_age: 7d
use_wal: true
6.2 Traefik Optimization
Update dynamic configuration:
http:
middlewares:
crowdsec:
plugin:
crowdsec:
updateIntervalSeconds: 10
defaultDecisionSeconds: 10
httpTimeoutSeconds: 5
Next Steps
- Implement AppSec WAF (Part 4)
- Configure captcha protection
- Set up alerts and notifications
- Monitor and tune performance
Part 4 Implementing CrowdSec AppSec WAF with Pangolin - Networking - HHF Technology Forums