# Need a second pair of eyes to solve a networking problem

**URL:** <https://forum.hhf.technology/t/need-a-second-pair-of-eyes-to-solve-a-networking-problem/4410>\
**Category:** Help\
**Tags:** traefik, pangolin, gerbil\
**Created:** [April 30, 2026, 3:00pm UTC](https://forum.hhf.technology/t/need-a-second-pair-of-eyes-to-solve-a-networking-problem/4410 "2026-04-30T15:00:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ovizii](https://forum.hhf.technology/user_avatar/forum.hhf.technology/ovizii/32/3371_2.png) [@ovizii](https://forum.hhf.technology/u/ovizii)\
**Post date:** [April 30, 2026, 3:00pm UTC](https://forum.hhf.technology/t/need-a-second-pair-of-eyes-to-solve-a-networking-problem/4410/1 "2026-04-30T15:00:24Z")

</div>

I read [this thread](https://forum.hhf.technology/t/securing-pangolin-newt-with-multi-network-docker-stacks/3833/18) and I think I got the gist, but I am facing a weird problem.

So my current compose file which contains pangolin, gerbil, traefik and crowdsec, works and has the following settings for networking. I removed anything unnecessary.

```auto
services:

  pangolin:
    image: docker.io/fosrl/pangolin:ee-1
    container_name: pangolin
    hostname: pangolin
    networks:
      pangolin:

  gerbil:
    image: docker.io/fosrl/gerbil:1.3.1
    container_name: gerbil
    hostname: gerbil
    command:
      - --reachableAt=http://gerbil:3004
      - --generateAndSaveKeyTo=/var/config/key
      - --remoteConfig=http://pangolin:3001/api/v1/
    ports:
      - 51820:51820/udp
      - 21820:21820/udp
      - 443:443
      - 80:80
    networks:
      pangolin:
      pangolin_adguardhome:
      pangolin_backrest:
      pangolin_bbs:
      pangolin_coroot:
      pangolin_drydock:
      pangolin_stalwart:
      pangolin_whoami:
      pangolin_xyops:

  traefik:
    image: traefik:v3
    container_name: traefik
    network_mode: service:gerbil # Ports appear on the gerbil service

```

My plan was to move traefik away from using `network_mode: service:gerbil`, so I modified my compose file as below with the result that everything works, except the sites on my other server running newt. It seems I have a newt/gerbil issue at hand somehow.

This is how I set up a public site, accessible via newt on a different server:

 ![image](https://forum-cdn.hhf.technology/original/2X/3/3797e783762a5c154b222ec5597f7910adc56a9a.png)

The remote newt’s logs seem perfectly fine.

```auto
❯ docker compose logs -f newt
newt | INFO: 2026/04/29 18:17:13 Newt version 1.12.2
newt | INFO: 2026/04/29 18:17:14 Server version: 1.17.1
newt | INFO: 2026/04/29 18:17:14 Websocket connected
newt | INFO: 2026/04/29 18:17:14 Connecting to endpoint: ctrl.foo.dj
newt | INFO: 2026/04/29 18:17:14 Tunnel connection to server established successfully!
newt | INFO: 2026/04/29 18:17:14 Started tcp proxy to network-tools:3000
newt | INFO: 2026/04/29 18:17:14 Started tcp proxy to backrest:9898
newt | INFO: 2026/04/29 18:17:14 Started tcp proxy to pocket-id:1411
newt | INFO: 2026/04/29 18:17:14 Started tcp proxy to 10.11.12.29:8123
newt | INFO: 2026/04/29 18:17:16 Client connectivity setup. Ready to accept connections from clients!

```

**New compose which leads to the above problem:**

```auto
services:

  pangolin:
    image: docker.io/fosrl/pangolin:ee-1
    container_name: pangolin
    hostname: pangolin
    networks:
      pangolin:

  gerbil:
    image: docker.io/fosrl/gerbil:1.3.1
    container_name: gerbil
    hostname: gerbil
    command:
      - --reachableAt=http://gerbil:3004
      - --generateAndSaveKeyTo=/var/config/key
      - --remoteConfig=http://pangolin:3001/api/v1/
    ports:
      - 51820:51820/udp
      - 21820:21820/udp
    networks:
      pangolin:

  traefik:
    image: traefik:v3
    container_name: traefik
    ports:
      - 443:443
      - 80:80
    networks:
      pangolin:
      pangolin_adguardhome:
      pangolin_backrest:
      pangolin_bbs:
      pangolin_coroot:
      pangolin_drydock:
      pangolin_stalwart:
      pangolin_whoami:
      pangolin_xyops:   

```

---

<div class="post-metadata">

**Author:** ![ovizii](https://forum.hhf.technology/user_avatar/forum.hhf.technology/ovizii/32/3371_2.png) [@ovizii](https://forum.hhf.technology/u/ovizii)\
**Post date:** [May 1, 2026, 6:31pm UTC](https://forum.hhf.technology/t/need-a-second-pair-of-eyes-to-solve-a-networking-problem/4410/2 "2026-05-01T18:31:59Z")

</div>

I have rechecked [pangolin’s docs](https://docs.pangolin.net/self-host/manual/docker-compose.md) and found out that `network_mode: service:gerbil` is actually the suggested usage. I never knew. I did that “workaround” when I couldn’t get stuff working without it.

I guess I’ll leave it at that.

---

<div class="post-metadata">

**Author:** ![hhf.technoloy](https://forum.hhf.technology/user_avatar/forum.hhf.technology/hhf.technoloy/32/68_2.png) [@hhf.technoloy](https://forum.hhf.technology/u/hhf.technoloy)\
**Post date:** [May 4, 2026, 4:15am UTC](https://forum.hhf.technology/t/need-a-second-pair-of-eyes-to-solve-a-networking-problem/4410/3 "2026-05-04T04:15:38Z")

</div>

**`network_mode: service:gerbil` is critical** in the standard Pangolin stack because **Gerbil owns the WireGuard network namespace** where all the tunneling magic (Newt peers) happens. Traefik must share that exact network stack to properly route traffic to remote Newt instances.

### Core Architecture (Pangolin + Gerbil + Newt + Traefik)

- **Pangolin** : The management/control plane (GUI, config, auth, site definitions). It talks to Gerbil and pushes dynamic config to Traefik.
- **Gerbil** : The data-plane tunnel server. It runs WireGuard, handles:
  - Incoming tunnels from remote **Newt** clients (UDP 51820 primarily).
  - Client connections (UDP 21820).
  - Peer management, key exchange, routing.

- **Traefik** : The actual L7 reverse proxy (with a Pangolin/Badger plugin/middleware for auth, routing rules, etc.).
- **Newt** (on remote servers): Lightweight WireGuard + TCP/UDP proxy client. It creates a secure tunnel back to Gerbil and exposes local services (or entire networks) into the Pangolin-controlled overlay.

**Traffic flow for a remote site (your case)**:

1. Public request → Gerbil’s exposed ports (80/443 on host).
2. Gerbil’s WireGuard interface receives the packet (or routes it).
3. Traefik (sharing Gerbil’s netns) sees the request, applies rules, and forwards it **over the WireGuard tunnel** to the correct Newt peer.
4. Newt on the remote server proxies it to the local service (e.g., `network-tools:3000`, `backrest:9898`, etc.).

### Why `network_mode: service:gerbil` Matters

`network_mode: service:gerbil` makes Traefik **join Gerbil’s network namespace** (netns). This means:

- Traefik and Gerbil share the **same network interfaces** , IP stack, routing table, and crucially — the **WireGuard interface** and all its peers.
- When Newt connects, Gerbil adds a peer route. Traefik can immediately reach that peer’s tunnel IP **without any extra routing tricks, iptables, or published ports between containers**.
- Ports 80/443 published on Gerbil appear on the host and are handled by Traefik inside the shared netns.

**Without it** (your modified compose):

- Traefik runs in its own netns (or attached only to Docker bridge networks like `pangolin` + others).
- It can reach Pangolin (via Docker DNS on the shared `pangolin` network).
- But it **cannot natively reach Newt peers** because those peers live only inside Gerbil’s WireGuard interface.
- Result: Remote Newt-tunneled sites break (or become unreliable), while local services on the same Docker networks might still work via bridge routing.

This is exactly what you’re seeing: local stuff might appear okay, but remote Newt sites fail.

### Additional Benefits of the Shared Netns

- **Seamless routing** to dynamic tunnel IPs (Newt peers get IPs from Gerbil’s WireGuard pool).
- **No port conflicts** or extra `expose`/`ports` needed between Traefik and Gerbil.
- **Consistent view** of the overlay network for proxying, health checks, etc.
- Easier integration with other tools (e.g., DNS resolvers, additional proxies) that also join Gerbil’s netns.

### Your Modified Setup and the Problem

In the new compose:

- Gerbil only has the `pangolin` network + its UDP ports.
- Traefik has many networks + its own published 80/443.
- Traefik can talk to Pangolin but lacks direct access to Gerbil’s WireGuard peers → Newt tunnels don’t route properly from Traefik’s perspective.
- Newt logs look fine because the tunnel itself establishes (to Gerbil), but the return/proxy path through Traefik is broken.

### Recommended Fixes / Best Practices

1. **Preferred (Standard)**: Keep `network_mode: service:gerbil` on Traefik. Publish ports **only on Gerbil**. Attach other services to their isolated networks as needed, and let Traefik reach them via the `pangolin` network (or additional shared networks).

2. **If you really want Traefik independent** :

3. **Multi-network segmentation** (as discussed in the forum thread): Keep apps on isolated networks, attach only what’s necessary to `pangolin` or Gerbil’s netns. Gerbil/Traefik act as the controlled gateway.

### Quick Verification Commands

On the host/server:

```bash
# Check network namespaces
ip netns list
docker inspect gerbil | grep -E 'NetworkMode|IPAddress'
docker inspect traefik | grep -E 'NetworkMode|IPAddress'

# From inside Traefik container (if using network_mode)
docker exec traefik ping <some_newt_peer_ip> # or wg show

root@hal-server-719189:~# docker inspect traefik | grep -E 'NetworkMode|IPAddress'
            "NetworkMode": "container:60e0677b872437a24af1e5de4746a42351cc0d40594cbb00467d29a82bf0b9bd",
root@hal-server-719189:~# docker inspect gerbil | grep -E 'NetworkMode|IPAddress'
            "NetworkMode": "pangolin",
                    "IPAddress": "172.20.0.48",

```

This shared netns pattern is a common Docker trick for tools that need low-level network access (proxies + VPNs, nginx + WireGuard, etc.). It’s not a bug — it’s intentional design for Pangolin’s tunneled reverse proxy architecture.

---

<div class="post-metadata">

**Author:** ![hhf.technoloy](https://forum.hhf.technology/user_avatar/forum.hhf.technology/hhf.technoloy/32/68_2.png) [@hhf.technoloy](https://forum.hhf.technology/u/hhf.technoloy)\
**Post date:** [May 4, 2026, 4:22am UTC](https://forum.hhf.technology/t/need-a-second-pair-of-eyes-to-solve-a-networking-problem/4410/4 "2026-05-04T04:22:36Z")

</div>

**Here is a practical (but more complex) example** of running Traefik independently while still routing traffic to Gerbil’s WireGuard peers.

### Option 1: Macvlan + Static Routes (Recommended Alternative)

This gives Traefik its own networks while allowing it to reach the WireGuard interface inside Gerbil’s network namespace.

#### Step 1: Create a Macvlan Network (for low-level routing)

```yaml
networks:
  pangolin_macvlan:
    driver: macvlan
    driver_opts:
      parent: eth0 # ← Change to your host's main interface (eth0, enp3s0, etc.)
    ipam:
      config:
        - subnet: 192.168.1.0/24 # Your LAN subnet
          gateway: 192.168.1.1
          ip_range: 192.168.1.100/28 # Small range for containers

```

#### Step 2: Updated Docker Compose (Key Changes)

```yaml
services:
  gerbil:
    image: docker.io/fosrl/gerbil:1.3.1
    container_name: gerbil
    hostname: gerbil
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    devices:
      - /dev/net/tun:/dev/net/tun
    command:
      - --reachableAt=http://gerbil:3004
      - --generateAndSaveKeyTo=/var/config/key
      - --remoteConfig=http://pangolin:3001/api/v1/
    ports:
      - 51820:51820/udp
      - 21820:21820/udp
    networks:
      - pangolin # For communication with Pangolin
      - pangolin_macvlan # Give Gerbil a real LAN IP

  traefik:
    image: traefik:v3
    container_name: traefik
    ports:
      - 443:443
      - 80:80
    networks:
      - pangolin
      - pangolin_macvlan
      - pangolin_adguardhome
      - pangolin_backrest
      # ... your other networks
    extra_hosts:
      - "gerbil:172.20.x.x" # Optional: Gerbil's Docker bridge IP
    cap_add:
      - NET_ADMIN # Needed for custom routing inside container

  pangolin:
    # ... unchanged
    networks:
      - pangolin

```

#### Step 3: Routing Setup (Critical Part)

After `docker compose up -d`, run these commands **on the host** or via a small helper container/sidecar:

```bash
# 1. Find Gerbil's network namespace
GERBIL_PID=$(docker inspect -f '{{.State.Pid}}' gerbil)
GERBIL_NS="ns-${GERBIL_PID}"

# 2. Enter Gerbil's namespace and note the WireGuard interface IP (usually wg0)
sudo ip netns exec ${GERBIL_NS} ip addr show wg0 # Look for the IP, e.g. 10.0.0.1

# 3. From Traefik container, add route to WireGuard subnet
docker exec -it traefik ip route add 10.0.0.0/24 via <gerbil_macvlan_ip> dev eth0

```

**Better approach — Persistent sidecar helper** (recommended):

Create a tiny routing sidecar:

```yaml
  routing-helper:
    image: alpine:latest
    container_name: routing-helper
    network_mode: "service:traefik" # Or attach to macvlan
    command: sh -c "
      apk add iproute2 iptables;
      while true; do
        ip route add 10.0.0.0/24 via $(getent hosts gerbil | awk '{print $1}') || true;
        sleep 30;
      done
    "
    cap_add:
      - NET_ADMIN

```

You may also need `iptables` masquerading or forwarding rules on the host or inside Gerbil:

```bash
# On host (one-time or in startup script)
sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE
sudo sysctl -w net.ipv4.ip_forward=1

```

### Option 2: Pure Policy Routing / ip rule (Advanced)

Use policy-based routing so traffic destined for Newt peer IPs from Traefik goes via Gerbil.

This requires:

- Giving both containers `NET_ADMIN`
- Creating custom routing tables
- Using `ip rule` to mark traffic based on source IP or fwmark

This is more fragile and harder to maintain.

### Why This Is More Complicated Than `network_mode: service:gerbil`

- **network\_mode** gives perfect, zero-config shared namespace (same interfaces, routes, ARP table).
- Macvlan/routing adds latency, potential asymmetric routing issues, firewall complexity, and more points of failure.
- Newt peer IPs are dynamic (assigned by Gerbil) — you need to keep routes updated.
- Debugging becomes much harder (`tcpdump` in two namespaces, asymmetric paths, etc.).

### Recommendation

**Strongly prefer keeping `network_mode: service:gerbil`** unless you have a very specific reason (e.g., running Traefik on a different host entirely, or needing Traefik to bind to multiple macvlans for different VLANs).

If you really need separation, the macvlan + routing-helper sidecar above is the most workable path for a single-host setup.

**Here is the advanced policy routing version** using `ip rule` + `fwmark` (firewall mark). This is more robust than simple static routes because it handles dynamic Newt peer IPs better and avoids some asymmetric routing issues.

### Why Policy Routing + fwmark?

- You mark packets originating from Traefik.
- A custom routing table + `ip rule` forces those marked packets to route via Gerbil’s IP (which has direct access to the WireGuard interface).
- This is cleaner for dynamic environments where Newt peers appear/disappear.

### Full docker-compose.yml

```yaml
networks:
  pangolin:
    driver: bridge

  # Optional macvlan if you want Traefik/Gerbil on real LAN IPs
  pangolin_macvlan:
    driver: macvlan
    driver_opts:
      parent: eth0 # ← CHANGE to your host interface
    ipam:
      config:
        - subnet: 192.168.1.0/24
          gateway: 192.168.1.1

services:
  pangolin:
    image: docker.io/fosrl/pangolin:ee-1.
    container_name: pangolin
    networks:
      - pangolin
    # volumes, env, etc.

  gerbil:
    image: docker.io/fosrl/gerbil:1.3.1
    container_name: gerbil
    hostname: gerbil
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    devices:
      - /dev/net/tun:/dev/net/tun
    command:
      - --reachableAt=http://gerbil:3004
      - --generateAndSaveKeyTo=/var/config/key
      - --remoteConfig=http://pangolin:3001/api/v1/
    ports:
      - "51820:51820/udp"
      - "21820:21820/udp"
    networks:
      - pangolin
      - pangolin_macvlan
    volumes:
      - gerbil_config:/var/config
    restart: unless-stopped

  traefik:
    image: traefik:v3
    container_name: traefik
    hostname: traefik
    ports:
      - "80:80"
      - "443:443"
    networks:
      - pangolin
      - pangolin_macvlan
      # Add all your app networks here: pangolin_adguardhome, etc.
    cap_add:
      - NET_ADMIN
    volumes:
      - ./traefik:/etc/traefik
      - traefik_certificates:/letsencrypt
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1 # Important for fwmark
    restart: unless-stopped
    depends_on:
      - gerbil

  # Advanced Policy Routing Helper
  policy-router:
    image: alpine:latest
    container_name: policy-router
    restart: unless-stopped
    network_mode: "service:traefik" # Shares Traefik's namespace
    cap_add:
      - NET_ADMIN
    command: /bin/sh -c '
      apk add --no-cache iproute2 iptables;
      echo "=== Policy Routing Helper Started ===";
      while true; do
        GERBIL_IP=$(getent hosts gerbil | awk "{print \$1}");
        if [-n "$GERBIL_IP"]; then
          # Mark packets from Traefik with 0x233 (example mark)
          iptables -t mangle -A OUTPUT -m mark --mark 0x0 -j MARK --set-mark 0x233 2>/dev/null || true;

          # Create custom routing table (if not exists)
          ip route add default via $GERBIL_IP dev eth0 table 233 2>/dev/null || true;

          # Policy rule: traffic with mark 0x233 uses table 233
          ip rule add fwmark 0x233 table 233 priority 1000 2>/dev/null || true;

          # Ensure we can reach WireGuard subnet (adjust range as needed)
          ip route add 10.0.0.0/24 via $GERBIL_IP dev eth0 table 233 2>/dev/null || true;
        fi
        sleep 20;
      done
    '
    depends_on:
      - traefik

```

### Volumes

```yaml
volumes:
  gerbil_config:
  traefik_certificates:

```

### One-time Host Setup (Run after first start)

```bash
sudo sysctl -w net.ipv4.ip_forward=1
sudo sysctl -w net.ipv4.conf.all.src_valid_mark=1

```

### Recommended Startup Script (`start-advanced.sh`)

```bash
#!/bin/bash
set -e

docker compose up -d

sleep 10

echo "Applying policy routing..."
docker exec traefik iptables -t mangle -A OUTPUT -j MARK --set-mark 0x233 || true

echo "Setup done. Check routing:"
docker exec traefik ip rule show
docker exec traefik ip route show table 233
docker exec traefik ip route

```

### How to Verify

```bash
# Inside Traefik
docker exec -it traefik ip rule show
docker exec -it traefik ip route show table 233
docker exec -it traefik ping <newt_peer_ip> # e.g. 10.0.0.45

# Check Gerbil WireGuard peers
docker exec -it gerbil wg show

```

### Pros & Cons of This Approach

**Pros** :

- More flexible than simple `ip route add`.
- Better handling of dynamic peers.
- Can be extended with more specific marks/rules.

**Cons** :

- Significantly more complex to debug.
- Still not as clean as `network_mode: service:gerbil`.
- Potential for routing loops or blackholing if misconfigured.
- Requires `NET_ADMIN` and sysctls.

* * *

**My honest recommendation** : Even with this advanced setup, the **simplest and most reliable** solution remains `network_mode: service:gerbil`.

---

<div class="post-metadata">

**Author:** ![system](https://forum.hhf.technology/user_avatar/forum.hhf.technology/system/32/6437_2.png) [@system](https://forum.hhf.technology/u/system)\
**Post date:** [November 2, 2026, 5:27am UTC](https://forum.hhf.technology/t/need-a-second-pair-of-eyes-to-solve-a-networking-problem/4410/5 "2026-11-02T05:27:09Z")

</div>

This topic has had no activity for 30 days. **It will close in 7 days** unless someone replies.

- Still broken? Reply with your current versions and logs, and it stays open.
- Fixed it? Post what worked and mark it as the solution, so the next person finds it.

---

<div class="post-metadata">

**Author:** ![system](https://forum.hhf.technology/user_avatar/forum.hhf.technology/system/32/6437_2.png) [@system](https://forum.hhf.technology/u/system)\
**Post date:** [November 9, 2026, 5:27am UTC](https://forum.hhf.technology/t/need-a-second-pair-of-eyes-to-solve-a-networking-problem/4410/6 "2026-11-09T05:27:11Z")

</div>

Closed on November 9, 2026, 5:27am UTC


