Dashboard stops updating after runtime error (traefik-log-dashboard discussion #52)

:speech_balloon: GitHub discussion traefik-log-dashboard #52 in Help, started by dlhall111 on GitHub

Since updating to 1.0.6 (including adding resource limits, as recommended) I’m seeing the following errors logged in the backend after a few hours of uptime:

2025/08/04 16:22:50 Panic in pollLoop: runtime error: slice bounds out of range [8700:5800]
2025/08/04 16:22:50 Panic in watchLoop: runtime error: slice bounds out of range [8700:5800]

Once these errors are thrown the dashboard no longer updates but the frontend/backend service continue to run and the dashboard is still accessible.

Simply restarting the backend service restores normal operations.

Note: I’m using Beszel for docker resource monitoring and don’t see any unusual resource usage from the backend service (or the frontend) at the timeframe that the runtime error is logged. Memory and CPU usage is consistent before/after the error is logged.


:speech_balloon: Join in here or on GitHub.

hhftechnology commented on GitHub:

try latest 1.0.7

dlhall111 commented on GitHub:

@​hhftechnology I updated earlier today and will monitor for new occurrences and reopen if needed. Thanks!

dlhall111 commented on GitHub:

@​hhftechnology it happened just a few minutes after I typed the above… :confused:

2025/08/05 15:53:59 Panic in pollLoop: runtime error: slice bounds out of range [:116588] with capacity 65536
2025/08/05 15:53:59 Panic in watchLoop: runtime error: slice bounds out of range [:116588] with capacity 65536

hhftechnology commented on GitHub:

share your compose file, OS version and traefik version

dlhall111 commented on GitHub:

@​hhftechnology here’s my compose (redacted where needed):

services:
  backend:
    image: ghcr.io/hhftechnology/traefik-log-dashboard-backend:latest
    container_name: log-dashboard-backend
    restart: unless-stopped
    volumes:
      - /path/traefik/logs:/logs:ro # Mount the Traefik logs directory
      - ./maxmind:/maxmind
    environment:
      - NODE_ENV=production
      - TRAEFIK_LOG_FILE=/logs/access.log
      - PORT=3002
      - USE_MAXMIND=true
      - MAXMIND_DB_PATH=/maxmind/GeoLite2-City.mmdb
      - MAXMIND_FALLBACK_ONLINE=true
      - MAXMIND_LICENSE_KEY=apikey
      - GOGC=50
      - GOMEMLIMIT=500MiB
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 256M
        reservations:
          cpus: '0.25'
          memory: 128M
    networks:
      - pangolin
  frontend:
    image: ghcr.io/hhftechnology/traefik-log-dashboard-frontend:latest
    container_name: log-dashboard-frontend
    restart: unless-stopped
    environment:
      - BACKEND_PORT=3002
    depends_on:
      - backend
    deploy:
      resources:
        limits:
          cpus: '0.25'
          memory: 128M
        reservations:
          cpus: '0.1'
          memory: 64M
    networks:
      - pangolin

networks:
  pangolin:
    name: pangolin
    external: true

This is running in Docker 28.3.3 on Debian 12.2.0 with Traefik 3.5.0.

hhftechnology commented on GitHub:

services:
  backend:
    image: ghcr.io/hhftechnology/traefik-log-dashboard-backend:1.0.7
    container_name: log-dashboard-backend
    restart: unless-stopped
    volumes:
      - /path/traefik/logs:/logs:ro # Mount the Traefik logs directory
      - ./maxmind:/maxmind
    environment:
      - TRAEFIK_LOG_FILE=/logs/access.log
      - USE_MAXMIND=true
      - MAXMIND_DB_PATH=/maxmind/GeoLite2-City.mmdb
      - MAXMIND_FALLBACK_ONLINE=true
      - GOGC=50
      - GOMEMLIMIT=500MiB
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 256M
        reservations:
          cpus: '0.25'
          memory: 128M

  frontend:
    image: ghcr.io/hhftechnology/traefik-log-dashboard-frontend:l1.0.7
    container_name: log-dashboard-frontend
    restart: unless-stopped
    depends_on:
      - backend
    deploy:
      resources:
        limits:
          cpus: '0.25'
          memory: 128M
        reservations:
          cpus: '0.1'
          memory: 64M

networks:
  pangolin:
    name: pangolin
    external: true

Use this as it is. In target user private ip of the VPS/VM

dlhall111 commented on GitHub:

@​hhftechnology I had to re-add the “networks: pangolin” lines to frontend/backend otherwise it wasn’t reachable + had to use “latest” tag since I couldn’t pull it using “1.0.7” tag (it throws “manifest unknown”), but am now running it as you suggested otherwise:

services:
  backend:
    image: ghcr.io/hhftechnology/traefik-log-dashboard-backend:latest
    container_name: log-dashboard-backend
    restart: unless-stopped
    volumes:
      - /path/traefik/logs:/logs:ro # Mount the Traefik logs directory
      - ./maxmind:/maxmind
    environment:
      - TRAEFIK_LOG_FILE=/logs/access.log
      - USE_MAXMIND=true
      - MAXMIND_DB_PATH=/maxmind/GeoLite2-City.mmdb
      - MAXMIND_FALLBACK_ONLINE=true
      - GOGC=50
      - GOMEMLIMIT=500MiB
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 256M
        reservations:
          cpus: '0.25'
          memory: 128M
    networks:
      - pangolin

  frontend:
    image: ghcr.io/hhftechnology/traefik-log-dashboard-frontend:latest
    container_name: log-dashboard-frontend
    restart: unless-stopped
    depends_on:
      - backend
    deploy:
      resources:
        limits:
          cpus: '0.25'
          memory: 128M
        reservations:
          cpus: '0.1'
          memory: 64M
    networks:
      - pangolin

networks:
  pangolin:
    name: pangolin
    external: true

dlhall111 commented on GitHub:

@​hhftechnology not sure if this is any way related to the original issue, but I just had a crash with this exception (using the above compose):

fatal error: concurrent map iteration and map write
goroutine 57 [running]:
reflect.mapiternext(0x4ad8a9?)
  /usr/local/go/src/runtime/map.go:1397 +0x13
reflect.(*MapIter).Next(0xc000344d68?)
  /usr/local/go/src/reflect/value.go:2005 +0x74
encoding/json.mapEncoder.encode({0x40?}, 0xc0004d0080, {0x81b580?, 0xc0003d4068?, 0x2?}, {0xc?, 0x0?})
  /usr/local/go/src/encoding/json/encode.go:745 +0x334
encoding/json.structEncoder.encode({{{0xc00059d308, 0x15, 0x23}, 0xc0000897d0, 0xc000089920}}, 0xc0004d0080, {0x8868a0?, 0xc0003d4000?, 0xc0004d0070?}, {0x0, ...})
  /usr/local/go/src/encoding/json/encode.go:704 +0x21e
encoding/json.ptrEncoder.encode({0xc0004d0070?}, 0xc0004d0080, {0x7f2360?, 0xc0004d0070?, 0xc0004d0070?}, {0x8?, 0x0?})
  /usr/local/go/src/encoding/json/encode.go:876 +0x23c
encoding/json.structEncoder.encode({{{0xc000119208, 0x4, 0x4}, 0xc00023c0c0, 0xc00023c150}}, 0xc0004d0080, {0x85d0e0?, 0xc0004d0040?, 0x40e61b?}, {0x0, ...})
  /usr/local/go/src/encoding/json/encode.go:704 +0x21e
encoding/json.(*encodeState).reflectValue(0xc0004d0080, {0x85d0e0?, 0xc0004d0040?, 0x40e9c5?}, {0x38?, 0x0?})
  /usr/local/go/src/encoding/json/encode.go:321 +0x73
encoding/json.(*encodeState).marshal(0x40e785?, {0x85d0e0?, 0xc0004d0040?}, {0x40?, 0xf6?})
  /usr/local/go/src/encoding/json/encode.go:297 +0xc5
encoding/json.Marshal({0x85d0e0, 0xc0004d0040})
  /usr/local/go/src/encoding/json/encode.go:163 +0xd0
main.(*WebSocketClient).sendMessage(0xc0002b4cb0, {{0x89e04d, 0x6}, {0x8921a0, 0xc0003dc008}, {0x0, 0x0}, 0xc0003d4000})
  /app/websocket.go:290 +0x10c
main.(*WebSocketClient).sendNewLogWithStats(_, {{0xc0004066d8, 0x18}, {0xc000464d60, 0x19}, {0xc0003f9800, 0xb}, {0xc000403438, 0x3}, {0xc00044eaf0, ...}, ...})
  /app/websocket.go:359 +0x310
main.(*WebSocketClient).WritePump(0xc0002b4cb0)
  /app/websocket.go:193 +0x745
main.(*WebSocketClient).Start.func1()
  /app/websocket.go:57 +0x49
created by main.(*WebSocketClient).Start in goroutine 55
  /app/websocket.go:49 +0x56
goroutine 1 [chan receive, 91 minutes]:
main.main()
  /app/main.go:138 +0xbc5
goroutine 6 [select, 91 minutes]:
github.com/patrickmn/go-cache.(*janitor).Run(0xc0000858d0, 0xc000286240)
  /go/pkg/mod/github.com/patrickmn/go-cache@​v2.1.0+incompatible/cache.go:1079 +0x7d
created by github.com/patrickmn/go-cache.runJanitor in goroutine 1
  /go/pkg/mod/github.com/patrickmn/go-cache@​v2.1.0+incompatible/cache.go:1099 +0xdb
goroutine 7 [select, 2 minutes]:
main.startRetryProcessor.func1()
  /app/geoLocation.go:585 +0x5a
created by main.startRetryProcessor in goroutine 1
  /app/geoLocation.go:583 +0x4c
goroutine 9 [syscall, 91 minutes]:
os/signal.signal_recv()
  /usr/local/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
  /usr/local/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1 …

dlhall111 commented on GitHub:

@​hhftechnology I am able to reproduce this crash, sort of. At the time the crash occurred I was navigating around Jellyseerr so I went back and did some more navigating around Jellyseerr while viewing the frontend page and after maybe 20-30 seconds of loading different things in Jellyseerr (which was of course causing a LOT of URL hits to spin through the live logging) it hit this same crash again.

My assumption is that this is an issue with updating the frontend when a lot of rapid logging activity is occurring. For example, loading just the home (Discover) page of Jellyseerr results in about 150 log entries (most of that is the image loading).

Edit: I cannot reproduce any crash if I’m not actively accessing the frontend when I’m loading/refreshing Jellyseerr.

hhftechnology commented on GitHub:

currently i am running the dashboard with 12000K entries. no crash v1.0.7

dlhall111 commented on GitHub:

@​hhftechnology I’m not referring to the total number of accumulated requests, I’m referring to a high frequency of requests in a short time. This app is running at 15K accumulated requests at the moment with no issue… but I bring up the frontend page and am viewing the live log and then open Jellyseerr in another browser that burst of 100-150 updates to the live log is what I see at the point of the crash occurring and that’s something I can consistently reproduce.

Let me know if you have any questions or need more info, or if you’d like a separate bug filed for this new crash scenario (haven’t seen the original panic errors since deploying the updated compose you suggested).

dlhall111 commented on GitHub:

@​hhftechnology just had another occurrence of the original issue:

2025/08/07 14:59:42 Panic in watchLoop: runtime error: slice bounds out of range [:112715] with capacity 65536
2025/08/07 14:59:42 Panic in pollLoop: runtime error: slice bounds out of range [:112715] with capacity 65536

15,532 requests accumulated with an uptime of a little over 12 hours.

hhftechnology commented on GitHub:

log ingest is only 1000, i now reduced to 500

dlhall111 commented on GitHub:

@​hhftechnology will you be updating the pre-built images with these latest changes? I’m using Package traefik-log-dashboard-backend · GitHub and Package traefik-log-dashboard-frontend · GitHub images at the moment but they haven’t gotten the latest updates it looks like.

I still see the original issue (panic error, no more updating) about once or twice a day and can still reproduce the crash issue when loading Jellyseerr while viewing the frontend.

dlhall111 replied in a thread on GitHub to dlhall111:

@​hhftechnology I switched from the GitHub image to Docker Hub image to get up-to-date but can still reproduce both the “concurrent map iteration and map write” crash (when refreshing Jellyseerr a few times) and the “Panic in watchLoop/Panic in pollLoop” runtime errors that cease all updates until the backend is restarted.

dlhall111 commented on GitHub:

@​hhftechnology I switched to using the OTLP config earlier today and haven’t had any occurrences of the runtime error or the backend crash when refreshing Jellyseerr since then (yay!). However, I’m no longer getting any size information in the logs and the “Data Transmitted” section in the header is always “0 B” of course. I’ve been looking at how to maybe get this added via changes to the Traefik config.yml but so far I’m not seeing it… any idea on how to get sizes added to the OTLP output? Thanks!

Edit: Unfortunately I was able to reproduce the “fatal error: concurrent map iteration and map write” backend crash while scrolling through the “Movies” section of Jellyseerr (where it’s loading tons of images very fast)… this is with the sampleRate set to “1.0” (I had started out with “0.1” and increased in stages to see how it performed). I’ll set sampleRate back down a bit and see if the crash still occurs.

AsDinFlames commented on GitHub:

Hey
i gonna face the same issue right now. I used the dockerhub container and since this version was updated 4 month ago i created also the docker images from repository. I cant make it past then 2.000 entries. The backend services crashed cause of the same “map”-reason. And yes it’s not crashing if the frontend is stopped.

My traefik+crowdsec handles like 20 services and it takes like 5-10minutes to get past 2.000 entries.

In another issue posted here it was recommended to use “sample rate” to 0.1 as a workaround to get not that much crashed but i dont know where to set it.

Is this an issue you will and can fix in the future? Since your dashboard is by far the only and best nice looking dashboard i found which is leightweight whats important for me.

plakun commented on GitHub:

Hello, have the same issue, use latest docker-compose base from github main-branch.
After start service collects initial data and after few seconds backend shows error:

2025/09/05 06:02:37 [OTLP] Converted span 'EntryPoint' to log entry: POST /api/v1/osquery/config 200 (5.03ms)
fatal error: concurrent map iteration and map write

goroutine 174 [running]:
reflect.mapiternext(0x4befc9?)
        /usr/local/go/src/runtime/map.go:1397 +0x13
reflect.(*MapIter).Next(0xc0004a4cf0?)
        /usr/local/go/src/reflect/value.go:2005 +0x74
encoding/json.mapEncoder.encode({0xc0004a4e50?}, 0xc000314040, {0xc9bec0?, 0xc0001e23d8?, 0x2?}, {0xa?, 0x0?})
        /usr/local/go/src/encoding/json/encode.go:745 +0x334
encoding/json.structEncoder.encode({{{0xc0000f2c08, 0x18, 0x23}, 0xc000250f90, 0xc000250fc0}}, 0xc000314040, {0xd89d80?, 0xc0001e23c0?, 0xc000314030?}, {0x0, ...})
        /usr/local/go/src/encoding/json/encode.go:704 +0x21e
encoding/json.ptrEncoder.encode({0xc000314030?}, 0xc000314040, {0xc407e0?, 0xc000314030?, 0xc000314030?}, {0x8?, 0x0?})
        /usr/local/go/src/encoding/json/encode.go:876 +0x23c
encoding/json.structEncoder.encode({{{0xc0000e2008, 0x4, 0x4}, 0xc00032f170, 0xc00032f1a0}}, 0xc000314040, {0xd25900?, 0xc000314000?, 0x411b1b?}, {0x0, ...})
        /usr/local/go/src/encoding/json/encode.go:704 +0x21e
encoding/json.(*encodeState).reflectValue(0xc000314040, {0xd25900?, 0xc000314000?, 0x411ec5?}, {0x38?, 0x0?})
        /usr/local/go/src/encoding/json/encode.go:321 +0x73
encoding/json.(*encodeState).marshal(0x411c85?, {0xd25900?, 0xc000314000?}, {0x5f?, 0x1?})
        /usr/local/go/src/encoding/json/encode.go:297 +0xc5
encoding/json.Marshal({0xd25900, 0xc000314000})
        /usr/local/go/src/encoding/json/encode.go:163 +0xd0
main.(*WebSocketClient).sendMessage(0xc000398000, {{0xda70f9, 0x6}, {0xd9a220, 0xc0002fcc88}, {0x0, 0x0}, 0xc0001e23c0})
        /app/websocket.go:290 +0x10c
main.(*WebSocketClient).sendNewLogWithStats(_, {{0xc000490738, 0x15}, {0xc000490750, 0x14}, {0xc0003fbd10, 0xd}, {0xc0003fbc68, 0x4}, {0xc0003fc9c0, ...}, ...})
        /app/websocket.go:359 +0x32a
main.(*WebSocketClient).WritePump(0xc000398000)
        /app/websocket.go:193 +0x745
main.(*WebSocketClient).Start.func1()
        /app/websocket.go:57 +0x49
created by main.(*WebSocketClient).Start in goroutine 172
        /app/websocket.go:49 +0x56

goroutine 1 [chan receive, 4 minutes]:
main.main()
        /app/main.go:169 +0x1398

goroutine 8 [select, 4 minutes]:
github.com/patrickmn/go-cache.(*janitor).Run(0xc0001e1e30, 0xc00039f580)
        /go/pkg/mod/github.com/patrickmn/go-cache@​v2.1.0+incompatible/cache.go:1079 +0x7d
created by github.com/patrickmn/go-cache.runJanitor in goroutine 1
        /go/pkg/mod/github.com/patrickmn/go-cache@​v2.1.0+incompatible/cache.go:1099 +0xdb

goroutine 9 [select, 1 minutes]:
main.startRetryProcessor.func1()
        /app/geoLocation.go:585 +0x5a
created by main.startRetryProcessor in goroutine 1
        /app/geoLocation.go:583 …

ovizii commented on GitHub:

Using image: ghcr.io/hhftechnology/traefik-log-dashboard-backend:latest and image: ghcr.io/hhftechnology/traefik-log-dashboard-frontend:latest I am getting something similar, the errors start after finally getting this app up and running for like 10 minutes or so, then it starts with:

log-dashboard-backend  | 2025/09/09 19:52:48 WebSocket clients tracking enabled
log-dashboard-backend  | 2025/09/09 19:52:49 Loading last 1000 log entries from /logs/traefik-access.log
log-dashboard-backend  | 2025/09/09 19:52:49 Starting background geo processing...
log-dashboard-backend  | 2025/09/09 19:52:49 Processed geo data for 6 IPs. 0 IPs remaining in queue.
.
.
.
log-dashboard-backend  | 2025/09/09 20:11:40 [WebSocket] Sending initial data to client 20250909-201140-172.16.34.3:36438
log-dashboard-backend  | 2025/09/09 20:11:40 [WebSocket] Sending initial stats to client 20250909-201140-172.16.34.3:36438
log-dashboard-backend  | [GIN] 2025/09/09 - 20:11:40 | 200 |     212.467µs |  100.104.120.16 | GET      "/ws"
log-dashboard-backend  | 2025/09/09 20:11:40 [WebSocket] Sending 1000 initial logs to client 20250909-201140-172.16.34.3:36438
log-dashboard-backend  | 2025/09/09 20:11:40 [WebSocket] Client 20250909-201140-172.16.34.3:36438 subscribed to log updates
log-dashboard-backend  | 2025/09/09 20:11:48 [WebSocket] Health check: 1 clients healthy
log-dashboard-backend  | 2025/09/09 20:12:18 [WebSocket] Health check: 1 clients healthy
log-dashboard-backend  | 2025/09/09 20:12:48 [WebSocket] Health check: 1 clients healthy
log-dashboard-backend  | 2025/09/09 20:13:18 [WebSocket] Health check: 1 clients healthy
log-dashboard-backend  | fatal error: concurrent map iteration and map write

followed by lots of this and more. Not sure how useful it is to post it all here.

og-dashboard-backend  | goroutine 99 [running]:
log-dashboard-backend  | reflect.mapiternext(0x4ad8a9?)
log-dashboard-backend  |        /usr/local/go/src/runtime/map.go:1397 +0x13
log-dashboard-backend  | reflect.(*MapIter).Next(0xc00055cd68?)
log-dashboard-backend  |        /usr/local/go/src/reflect/value.go:2005 +0x74
log-dashboard-backend  | encoding/json.mapEncoder.encode({0xc0000121e0?}, 0xc00024a040, {0x81b580?, 0xc000310018?, 0x2?}, {0xa?, 0x0?})
log-dashboard-backend  |        /usr/local/go/src/encoding/json/encode.go:745 +0x334
log-dashboard-backend  | encoding/json.structEncoder.encode({{{0xc000222008, 0x15, 0x23}, 0xc0004485d0, 0xc000448600}}, 0xc00024a040, {0x8868a0?, 0xc000310000?, 0xc00024a0b0?}, {0x0, ...})
log-dashboard-backend  |        /usr/local/go/src/encoding/json/encode.go:704 +0x21e
log-dashboard-backend  | encoding/json.ptrEncoder.encode({0xc00024a0b0?}, 0xc00024a040, {0x7f2360?, 0xc00024a0b0?, 0xc00024a0b0?}, {0x8?, 0x0?})
log-dashboard-backend  |        /usr/local/go/src/encoding/json/encode.go:876 +0x23c
log-dashboard-backend  | encoding/json.structEncoder.encode({{{0xc000164908, 0x4, 0x4}, 0xc000448690, 0xc0004486c0}}, 0xc00024a040, {0x85d0e0?, …