Need a second pair of eyes to solve a networking problem

I read this thread and I think I got the gist, but I am facing a weird problem.

So my current compose file which contains pangolin, gerbil, traefik and crowdsec, works and has the following settings for networking. I removed anything unnecessary.

services:

  pangolin:
    image: docker.io/fosrl/pangolin:ee-1
    container_name: pangolin
    hostname: pangolin
    networks:
      pangolin:

  gerbil:
    image: docker.io/fosrl/gerbil:1.3.1
    container_name: gerbil
    hostname: gerbil
    command:
      - --reachableAt=http://gerbil:3004
      - --generateAndSaveKeyTo=/var/config/key
      - --remoteConfig=http://pangolin:3001/api/v1/
    ports:
      - 51820:51820/udp
      - 21820:21820/udp
      - 443:443
      - 80:80
    networks:
      pangolin:
      pangolin_adguardhome:
      pangolin_backrest:
      pangolin_bbs:
      pangolin_coroot:
      pangolin_drydock:
      pangolin_stalwart:
      pangolin_whoami:
      pangolin_xyops:

  traefik:
    image: traefik:v3
    container_name: traefik
    network_mode: service:gerbil # Ports appear on the gerbil service

My plan was to move traefik away from using network_mode: service:gerbil, so I modified my compose file as below with the result that everything works, except the sites on my other server running newt. It seems I have a newt/gerbil issue at hand somehow.

This is how I set up a public site, accessible via newt on a different server:

The remote newt’s logs seem perfectly fine.

❯ docker compose logs -f newt
newt  | INFO: 2026/04/29 18:17:13 Newt version 1.12.2
newt  | INFO: 2026/04/29 18:17:14 Server version: 1.17.1
newt  | INFO: 2026/04/29 18:17:14 Websocket connected
newt  | INFO: 2026/04/29 18:17:14 Connecting to endpoint: ctrl.foo.dj
newt  | INFO: 2026/04/29 18:17:14 Tunnel connection to server established successfully!
newt  | INFO: 2026/04/29 18:17:14 Started tcp proxy to network-tools:3000
newt  | INFO: 2026/04/29 18:17:14 Started tcp proxy to backrest:9898
newt  | INFO: 2026/04/29 18:17:14 Started tcp proxy to pocket-id:1411
newt  | INFO: 2026/04/29 18:17:14 Started tcp proxy to 10.11.12.29:8123
newt  | INFO: 2026/04/29 18:17:16 Client connectivity setup. Ready to accept connections from clients!

New compose which leads to the above problem:

services:

  pangolin:
    image: docker.io/fosrl/pangolin:ee-1
    container_name: pangolin
    hostname: pangolin
    networks:
      pangolin:

  gerbil:
    image: docker.io/fosrl/gerbil:1.3.1
    container_name: gerbil
    hostname: gerbil
    command:
      - --reachableAt=http://gerbil:3004
      - --generateAndSaveKeyTo=/var/config/key
      - --remoteConfig=http://pangolin:3001/api/v1/
    ports:
      - 51820:51820/udp
      - 21820:21820/udp
    networks:
      pangolin:

  traefik:
    image: traefik:v3
    container_name: traefik
    ports:
      - 443:443
      - 80:80
    networks:
      pangolin:
      pangolin_adguardhome:
      pangolin_backrest:
      pangolin_bbs:
      pangolin_coroot:
      pangolin_drydock:
      pangolin_stalwart:
      pangolin_whoami:
      pangolin_xyops:   

I have rechecked pangolin’s docs and found out that network_mode: service:gerbil is actually the suggested usage. I never knew. I did that “workaround” when I couldn’t get stuff working without it.

I guess I’ll leave it at that.

network_mode: service:gerbil is critical in the standard Pangolin stack because Gerbil owns the WireGuard network namespace where all the tunneling magic (Newt peers) happens. Traefik must share that exact network stack to properly route traffic to remote Newt instances.

Core Architecture (Pangolin + Gerbil + Newt + Traefik)

  • Pangolin: The management/control plane (GUI, config, auth, site definitions). It talks to Gerbil and pushes dynamic config to Traefik.
  • Gerbil: The data-plane tunnel server. It runs WireGuard, handles:
    • Incoming tunnels from remote Newt clients (UDP 51820 primarily).
    • Client connections (UDP 21820).
    • Peer management, key exchange, routing.
  • Traefik: The actual L7 reverse proxy (with a Pangolin/Badger plugin/middleware for auth, routing rules, etc.).
  • Newt (on remote servers): Lightweight WireGuard + TCP/UDP proxy client. It creates a secure tunnel back to Gerbil and exposes local services (or entire networks) into the Pangolin-controlled overlay.

Traffic flow for a remote site (your case):

  1. Public request → Gerbil’s exposed ports (80/443 on host).
  2. Gerbil’s WireGuard interface receives the packet (or routes it).
  3. Traefik (sharing Gerbil’s netns) sees the request, applies rules, and forwards it over the WireGuard tunnel to the correct Newt peer.
  4. Newt on the remote server proxies it to the local service (e.g., network-tools:3000, backrest:9898, etc.).

Why network_mode: service:gerbil Matters

network_mode: service:gerbil makes Traefik join Gerbil’s network namespace (netns). This means:

  • Traefik and Gerbil share the same network interfaces, IP stack, routing table, and crucially — the WireGuard interface and all its peers.
  • When Newt connects, Gerbil adds a peer route. Traefik can immediately reach that peer’s tunnel IP without any extra routing tricks, iptables, or published ports between containers.
  • Ports 80/443 published on Gerbil appear on the host and are handled by Traefik inside the shared netns.

Without it (your modified compose):

  • Traefik runs in its own netns (or attached only to Docker bridge networks like pangolin + others).
  • It can reach Pangolin (via Docker DNS on the shared pangolin network).
  • But it cannot natively reach Newt peers because those peers live only inside Gerbil’s WireGuard interface.
  • Result: Remote Newt-tunneled sites break (or become unreliable), while local services on the same Docker networks might still work via bridge routing.

This is exactly what you’re seeing: local stuff might appear okay, but remote Newt sites fail.

Additional Benefits of the Shared Netns

  • Seamless routing to dynamic tunnel IPs (Newt peers get IPs from Gerbil’s WireGuard pool).
  • No port conflicts or extra expose/ports needed between Traefik and Gerbil.
  • Consistent view of the overlay network for proxying, health checks, etc.
  • Easier integration with other tools (e.g., DNS resolvers, additional proxies) that also join Gerbil’s netns.

Your Modified Setup and the Problem

In the new compose:

  • Gerbil only has the pangolin network + its UDP ports.
  • Traefik has many networks + its own published 80/443.
  • Traefik can talk to Pangolin but lacks direct access to Gerbil’s WireGuard peers → Newt tunnels don’t route properly from Traefik’s perspective.
  • Newt logs look fine because the tunnel itself establishes (to Gerbil), but the return/proxy path through Traefik is broken.

Recommended Fixes / Best Practices

  1. Preferred (Standard): Keep network_mode: service:gerbil on Traefik. Publish ports only on Gerbil. Attach other services to their isolated networks as needed, and let Traefik reach them via the pangolin network (or additional shared networks).

  2. If you really want Traefik independent:

    • Run a sidecar or additional routing (e.g., route traffic from Traefik’s netns to Gerbil’s via a macvlan or custom routing).
    • Or attach Traefik to Gerbil via a shared network and handle WireGuard routing manually (complex, error-prone, not recommended).
    • Use Gerbil purely for tunneling and have Traefik forward to Gerbil’s service IP, but this adds latency and complexity for Newt routing.
  3. Multi-network segmentation (as discussed in the forum thread): Keep apps on isolated networks, attach only what’s necessary to pangolin or Gerbil’s netns. Gerbil/Traefik act as the controlled gateway.

Quick Verification Commands

On the host/server:

# Check network namespaces
ip netns list
docker inspect gerbil | grep -E 'NetworkMode|IPAddress'
docker inspect traefik | grep -E 'NetworkMode|IPAddress'

# From inside Traefik container (if using network_mode)
docker exec traefik ping <some_newt_peer_ip>   # or wg show

root@hal-server-719189:~# docker inspect traefik | grep -E 'NetworkMode|IPAddress'
            "NetworkMode": "container:60e0677b872437a24af1e5de4746a42351cc0d40594cbb00467d29a82bf0b9bd",
root@hal-server-719189:~# docker inspect gerbil | grep -E 'NetworkMode|IPAddress'
            "NetworkMode": "pangolin",
                    "IPAddress": "172.20.0.48",

This shared netns pattern is a common Docker trick for tools that need low-level network access (proxies + VPNs, nginx + WireGuard, etc.). It’s not a bug — it’s intentional design for Pangolin’s tunneled reverse proxy architecture.

Here is a practical (but more complex) example of running Traefik independently while still routing traffic to Gerbil’s WireGuard peers.

Option 1: Macvlan + Static Routes (Recommended Alternative)

This gives Traefik its own networks while allowing it to reach the WireGuard interface inside Gerbil’s network namespace.

Step 1: Create a Macvlan Network (for low-level routing)

networks:
  pangolin_macvlan:
    driver: macvlan
    driver_opts:
      parent: eth0                  # ← Change to your host's main interface (eth0, enp3s0, etc.)
    ipam:
      config:
        - subnet: 192.168.1.0/24    # Your LAN subnet
          gateway: 192.168.1.1
          ip_range: 192.168.1.100/28  # Small range for containers

Step 2: Updated Docker Compose (Key Changes)

services:
  gerbil:
    image: docker.io/fosrl/gerbil:1.3.1
    container_name: gerbil
    hostname: gerbil
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    devices:
      - /dev/net/tun:/dev/net/tun
    command:
      - --reachableAt=http://gerbil:3004
      - --generateAndSaveKeyTo=/var/config/key
      - --remoteConfig=http://pangolin:3001/api/v1/
    ports:
      - 51820:51820/udp
      - 21820:21820/udp
    networks:
      - pangolin          # For communication with Pangolin
      - pangolin_macvlan  # Give Gerbil a real LAN IP

  traefik:
    image: traefik:v3
    container_name: traefik
    ports:
      - 443:443
      - 80:80
    networks:
      - pangolin
      - pangolin_macvlan
      - pangolin_adguardhome
      - pangolin_backrest
      # ... your other networks
    extra_hosts:
      - "gerbil:172.20.x.x"   # Optional: Gerbil's Docker bridge IP
    cap_add:
      - NET_ADMIN             # Needed for custom routing inside container

  pangolin:
    # ... unchanged
    networks:
      - pangolin

Step 3: Routing Setup (Critical Part)

After docker compose up -d, run these commands on the host or via a small helper container/sidecar:

# 1. Find Gerbil's network namespace
GERBIL_PID=$(docker inspect -f '{{.State.Pid}}' gerbil)
GERBIL_NS="ns-${GERBIL_PID}"

# 2. Enter Gerbil's namespace and note the WireGuard interface IP (usually wg0)
sudo ip netns exec ${GERBIL_NS} ip addr show wg0   # Look for the IP, e.g. 10.0.0.1

# 3. From Traefik container, add route to WireGuard subnet
docker exec -it traefik ip route add 10.0.0.0/24 via <gerbil_macvlan_ip> dev eth0

Better approach — Persistent sidecar helper (recommended):

Create a tiny routing sidecar:

  routing-helper:
    image: alpine:latest
    container_name: routing-helper
    network_mode: "service:traefik"   # Or attach to macvlan
    command: sh -c "
      apk add iproute2 iptables;
      while true; do
        ip route add 10.0.0.0/24 via $(getent hosts gerbil | awk '{print $1}') || true;
        sleep 30;
      done
    "
    cap_add:
      - NET_ADMIN

You may also need iptables masquerading or forwarding rules on the host or inside Gerbil:

# On host (one-time or in startup script)
sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE
sudo sysctl -w net.ipv4.ip_forward=1

Option 2: Pure Policy Routing / ip rule (Advanced)

Use policy-based routing so traffic destined for Newt peer IPs from Traefik goes via Gerbil.

This requires:

  • Giving both containers NET_ADMIN
  • Creating custom routing tables
  • Using ip rule to mark traffic based on source IP or fwmark

This is more fragile and harder to maintain.

Why This Is More Complicated Than network_mode: service:gerbil

  • network_mode gives perfect, zero-config shared namespace (same interfaces, routes, ARP table).
  • Macvlan/routing adds latency, potential asymmetric routing issues, firewall complexity, and more points of failure.
  • Newt peer IPs are dynamic (assigned by Gerbil) — you need to keep routes updated.
  • Debugging becomes much harder (tcpdump in two namespaces, asymmetric paths, etc.).

Recommendation

Strongly prefer keeping network_mode: service:gerbil unless you have a very specific reason (e.g., running Traefik on a different host entirely, or needing Traefik to bind to multiple macvlans for different VLANs).

If you really need separation, the macvlan + routing-helper sidecar above is the most workable path for a single-host setup.

Here is the advanced policy routing version using ip rule + fwmark (firewall mark). This is more robust than simple static routes because it handles dynamic Newt peer IPs better and avoids some asymmetric routing issues.

Why Policy Routing + fwmark?

  • You mark packets originating from Traefik.
  • A custom routing table + ip rule forces those marked packets to route via Gerbil’s IP (which has direct access to the WireGuard interface).
  • This is cleaner for dynamic environments where Newt peers appear/disappear.

Full docker-compose.yml

networks:
  pangolin:
    driver: bridge

  # Optional macvlan if you want Traefik/Gerbil on real LAN IPs
  pangolin_macvlan:
    driver: macvlan
    driver_opts:
      parent: eth0                    # ← CHANGE to your host interface
    ipam:
      config:
        - subnet: 192.168.1.0/24
          gateway: 192.168.1.1

services:
  pangolin:
    image: docker.io/fosrl/pangolin:ee-1.
    container_name: pangolin
    networks:
      - pangolin
    # volumes, env, etc.

  gerbil:
    image: docker.io/fosrl/gerbil:1.3.1
    container_name: gerbil
    hostname: gerbil
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    devices:
      - /dev/net/tun:/dev/net/tun
    command:
      - --reachableAt=http://gerbil:3004
      - --generateAndSaveKeyTo=/var/config/key
      - --remoteConfig=http://pangolin:3001/api/v1/
    ports:
      - "51820:51820/udp"
      - "21820:21820/udp"
    networks:
      - pangolin
      - pangolin_macvlan
    volumes:
      - gerbil_config:/var/config
    restart: unless-stopped

  traefik:
    image: traefik:v3
    container_name: traefik
    hostname: traefik
    ports:
      - "80:80"
      - "443:443"
    networks:
      - pangolin
      - pangolin_macvlan
      # Add all your app networks here: pangolin_adguardhome, etc.
    cap_add:
      - NET_ADMIN
    volumes:
      - ./traefik:/etc/traefik
      - traefik_certificates:/letsencrypt
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1   # Important for fwmark
    restart: unless-stopped
    depends_on:
      - gerbil

  # Advanced Policy Routing Helper
  policy-router:
    image: alpine:latest
    container_name: policy-router
    restart: unless-stopped
    network_mode: "service:traefik"   # Shares Traefik's namespace
    cap_add:
      - NET_ADMIN
    command: /bin/sh -c '
      apk add --no-cache iproute2 iptables;
      echo "=== Policy Routing Helper Started ===";
      while true; do
        GERBIL_IP=$(getent hosts gerbil | awk "{print \$1}");
        if [ -n "$GERBIL_IP" ]; then
          # Mark packets from Traefik with 0x233 (example mark)
          iptables -t mangle -A OUTPUT -m mark --mark 0x0 -j MARK --set-mark 0x233 2>/dev/null || true;

          # Create custom routing table (if not exists)
          ip route add default via $GERBIL_IP dev eth0 table 233 2>/dev/null || true;

          # Policy rule: traffic with mark 0x233 uses table 233
          ip rule add fwmark 0x233 table 233 priority 1000 2>/dev/null || true;

          # Ensure we can reach WireGuard subnet (adjust range as needed)
          ip route add 10.0.0.0/24 via $GERBIL_IP dev eth0 table 233 2>/dev/null || true;
        fi
        sleep 20;
      done
    '
    depends_on:
      - traefik

Volumes

volumes:
  gerbil_config:
  traefik_certificates:

One-time Host Setup (Run after first start)

sudo sysctl -w net.ipv4.ip_forward=1
sudo sysctl -w net.ipv4.conf.all.src_valid_mark=1

Recommended Startup Script (start-advanced.sh)

#!/bin/bash
set -e

docker compose up -d

sleep 10

echo "Applying policy routing..."
docker exec traefik iptables -t mangle -A OUTPUT -j MARK --set-mark 0x233 || true

echo "Setup done. Check routing:"
docker exec traefik ip rule show
docker exec traefik ip route show table 233
docker exec traefik ip route

How to Verify

# Inside Traefik
docker exec -it traefik ip rule show
docker exec -it traefik ip route show table 233
docker exec -it traefik ping <newt_peer_ip>   # e.g. 10.0.0.45

# Check Gerbil WireGuard peers
docker exec -it gerbil wg show

Pros & Cons of This Approach

Pros:

  • More flexible than simple ip route add.
  • Better handling of dynamic peers.
  • Can be extended with more specific marks/rules.

Cons:

  • Significantly more complex to debug.
  • Still not as clean as network_mode: service:gerbil.
  • Potential for routing loops or blackholing if misconfigured.
  • Requires NET_ADMIN and sysctls.

My honest recommendation: Even with this advanced setup, the simplest and most reliable solution remains network_mode: service:gerbil.