Backend claims all memory and crashes (traefik-log-dashboard discussion #30)

:speech_balloon: GitHub discussion traefik-log-dashboard #30 in Help, started by EmilyxFox on GitHub

This is kind of strange and I haven’t dug super deep into it but here goes.

It seems like the new (1.0.4) backend eats about 5GB of ram a second until it crashes, only to restart and do the same again over and over again. My log file is about 1MB. I have ~20GB free on the system and it takes about 4-5 seconds from container start to container exit (137). I’m not entirely sure what could be causing this, here is my compose file:

dashboard-backend:
    image: ghcr.io/hhftechnology/traefik-log-dashboard-backend:1.0.5@​sha256:dee2ef4d2ffd074ca107d765db74336b11c38b99b62aecbb9c2d971b56e20f11
    environment:
      NODE_ENV: production
      PORT: 3001
      TRAEFIK_LOG_FILE: /logs/access.log
    volumes:
      - /mnt/ssd/docker/traefik/logs:/logs:ro
    networks:
      - dashboard
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:3001/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  dashboard-frontend:
    image: ghcr.io/hhftechnology/traefik-log-dashboard-frontend:1.0.5@​sha256:b771be208f4c7b5c8271f2e6c8ce73539dd14d665e0cbb231934c6aeddd84e81
    environment:
      BACKEND_SERVICE: dashboard-backend
      BACKEND_PORT: 3001
    ports:
      - "3000:80"
    networks:
      - dashboard
    depends_on:
      - dashboard-backend
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost/"]
      interval: 30s …

---
💬 Join in here or [on GitHub](https://github.com/hhftechnology/traefik-log-dashboard/discussions/30).
<!-- hhf-forum-sync -->

BingoRox commented on GitHub:

Yea this completely crashed my VPS multiple times, I haven’t had a chance to try and debug but this is pretty bad I just removed the containers for now.

Infamousbugg commented on GitHub:

I had the same issue, so I just went back to 1.0.3. Really looking forward to HA Pangolin.

lekorrigan commented on GitHub:

Same, and I was also running out of disk space somehow (admittedly on a tiny VPS with 10GB of disk space)

nesalkiN commented on GitHub:

Yes. Same. My VPS became very unhappy until docker compose down finally did something.
The missing parts in the graph is forced reboots when it stopped responding at all.

tomorrow-nf commented on GitHub:

Seeing the same here

vcdx71 commented on GitHub:

Yes seeing this as well.

Node815 commented on GitHub:

I had the same error and rolled mine back to 1.0.3 for now too.

lostb1t commented on GitHub:

yup, get OOM killed very fast. Gave it 2000mb

Node815 commented on GitHub:

Updated to 1.0.5 and it also does the same memory issue. Might be best to watch for any updates in the change log for this issue so we know it’s safe to update.

hhftechnology commented on GitHub:

  backend:
    image: ghcr.io/hhftechnology/traefik-log-dashboard-backend:1.0.5
    container_name: log-dashboard-backend
    restart: unless-stopped
    volumes:
      - ./config/traefik/logs:/logs:ro # Mount the Traefik logs directory
      - ./config/maxmind:/maxmind # Mount the Traefik logs directory
    environment:
      - PORT=3001
      - TRAEFIK_LOG_FILE=/logs/access.log
      - USE_MAXMIND=true
      - MAXMIND_DB_PATH=/maxmind/GeoLite2-City.mmdb
      - MAXMIND_FALLBACK_ONLINE=true
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 256M
        reservations:
          cpus: '0.1'
          memory: 64M

  frontend:
    image: ghcr.io/hhftechnology/traefik-log-dashboard-frontend:1.0.5
    container_name: log-dashboard-frontend
    restart: unless-stopped
    ports:
      - "3000:80"
    depends_on:
      - backend
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 256M
        reservations:
          cpus: '0.1'
          memory: 64M

get the maxmind file from the repo https://github.com/hhftechnology/traefik-log-dashboard/tree/main/maxmind

hhftechnology commented on GitHub:

1.0.5 without limits

hhftechnology commented on GitHub:

Let me know if you if you still get high memory
@​lostb1t @​nesalkiN @​tomorrow-nf @​vcdx71 @​Node815

Razdnut commented on GitHub:

@​hhftechnology Thank you. It would be possible to have a more understandable README in QuickStart when integrating MaxMind because now it is divided into multiple parts that do different things. I wouldn’t want the problem to arise from a wrong configuration we set up because I personally started from:

docker compose down
docker compose build --no-cache
docker compose up -d

going to supplement MaxMind’s variables. :blush:
This is considering that on your configuration you had no problems.

hhftechnology commented on GitHub:

@​hhftechnology Thank you. It would be possible to have a more understandable README in QuickStart when integrating MaxMind because now it is divided into multiple parts that do different things. I wouldn’t want the problem to arise from a wrong configuration we set up because I personally started from:

docker compose down
docker compose build --no-cache
docker compose up -d
going to supplement MaxMind’s variables. :blush: This is considering that on your configuration you had no problems.

yes i would. but for future always cap your resources when you deploy a log viewer/analytic container.

lostb1t commented on GitHub:

maybe bundle the geo city db with the docker image? its not that big

Razdnut commented on GitHub:

@​hhftechnology I text the new compose but it crashes.

I think it goes in out-of-memory (OOM) killer.

Can it be related to the fact that my access.logs is 1.3 GB?

GIF:

My compose:

Details

services:
  backend:
    build: ./backend
    container_name: ${BACKEND_CONTAINER_NAME:-traefik-dashboard-backend}
    environment:
      - NODE_ENV=production
      - PORT=3001
      - TRAEFIK_LOG_FILE=/logs/access.log
      - USE_MAXMIND=true
      - MAXMIND_DB_PATH=/maxmind/GeoLite2-City.mmdb
      - MAXMIND_FALLBACK_ONLINE=true
    volumes:
      # Mount your Traefik log file or directory here
      - ${TRAEFIK_LOG_PATH:-./logs}:/logs:ro
      - ./traefik-log-dashboard/maxmind:/maxmind # Mount the Maxmind database directory
    ports:
      - "3001:3001"
    restart: unless-stopped
    # healthcheck:
      # test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:3001/health"]
      # interval: 30s
      # timeout: 10s
      # retries: 3
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 1500M
        reservations:
          cpus: '0.1'
          memory: 64M

  frontend:
    build: ./frontend
    container_name: ${FRONTEND_CONTAINER_NAME:-traefik-dashboard-frontend}
    environment:
      - BACKEND_SERVICE=${BACKEND_SERVICE_NAME:-backend}
      - BACKEND_PORT=${BACKEND_PORT:-3001}
    ports:
      - "3000:80"
    depends_on:
      - backend
    restart: unless-stopped
    # healthcheck:
      # test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost/"]
      # interval: 30s
      # timeout: 10s
      # retries: 3
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 1500M
        reservations:
          cpus: '0.1'
          memory: 64M

My ENV:

Details

# Path to your Traefik log file or directory
# Can be a single path or comma-separated list of paths
# Examples:
# - Single file: /path/to/traefik.log
# - Single directory: /path/to/logs/
# - Multiple paths: /path/to/logs1/,/path/to/logs2/,/path/to/specific.log
TRAEFIK_LOG_PATH=/Volume1/pangolin/config/traefik/logs

# Backend API port (optional, default: 3001)
PORT=3001

# Frontend port (optional, default: 3000)
FRONTEND_PORT=3000

# Backend service name for Docker networking (optional, default: backend)
BACKEND_SERVICE_NAME=backend

# Container names (optional, with defaults)
BACKEND_CONTAINER_NAME=traefik-dashboard-backend
FRONTEND_CONTAINER_NAME=traefik-dashboard-frontend

Logs:

Details

traefik-dashboard-backend   | [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
traefik-dashboard-backend   |
traefik-dashboard-backend   | [GIN-debug] [WARNING] Running in "debug" mode. Switch to …

EmilyxFox commented on GitHub:

  dashboard-backend:
    image: ghcr.io/hhftechnology/traefik-log-dashboard-backend:1.0.5@​sha256:dee2ef4d2ffd074ca107d765db74336b11c38b99b62aecbb9c2d971b56e20f11
    environment:
      - PORT=3001
      - TRAEFIK_LOG_FILE=/logs/access.log
      - USE_MAXMIND=true
      - MAXMIND_DB_PATH=/maxmind/GeoLite2-City.mmdb
      - MAXMIND_FALLBACK_ONLINE=true
    volumes:
      - /mnt/ssd/docker/traefik/logs:/logs:ro
      - ./dashboard/maxmind:/maxmind
    networks:
      - dashboard
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:3001/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 256M
        reservations:
          cpus: '0.1'
          memory: 64M

  dashboard-frontend:
    image: ghcr.io/hhftechnology/traefik-log-dashboard-frontend:1.0.5@​sha256:b771be208f4c7b5c8271f2e6c8ce73539dd14d665e0cbb231934c6aeddd84e81
    environment:
      - BACKEND_SERVICE=dashboard-backend
      - BACKEND_PORT=3001
    ports:
      - "3000:80"
    networks:
      - dashboard
    depends_on:
      - dashboard-backend
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost/"]
      interval: 30s
      timeout: 10s
      retries: 3
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 256M
        reservations:
          cpus: '0.1'
          memory: 64M

I’ve made these changes

  • changed env syntax
  • added maxmind env vars
  • set resource limits
  • make the maxmind database file available to the container (I shelled in to check it was there)

Its still acting in the exact same way. Runs for a couple of seconds with very high CPU usage (if no limit) and then restarts

hhftechnology commented on GitHub:

dashboard-backend:
image: ghcr.io/hhftechnology/traefik-log-dashboard-backend:1.0.5@​sha256:dee2ef4d2ffd074ca107d765db74336b11c38b99b62aecbb9c2d971b56e20f11
environment:
- PORT=3001
- TRAEFIK_LOG_FILE=/logs/access.log
- USE_MAXMIND=true
- MAXMIND_DB_PATH=/maxmind/GeoLite2-City.mmdb
- MAXMIND_FALLBACK_ONLINE=true
volumes:
- /mnt/ssd/docker/traefik/logs:/logs:ro
- ./dashboard/maxmind:/maxmind
networks:
- dashboard
restart: unless-stopped
healthcheck:
test: [“CMD”, “wget”, “–quiet”, “–tries=1”, “–spider”, “http://localhost:3001/health”]
interval: 30s
timeout: 10s
retries: 3
deploy:
resources:
limits:
cpus: ‘0.5’
memory: 256M
reservations:
cpus: ‘0.1’
memory: 64M

dashboard-frontend:
image: ghcr.io/hhftechnology/traefik-log-dashboard-frontend:1.0.5@​sha256:b771be208f4c7b5c8271f2e6c8ce73539dd14d665e0cbb231934c6aeddd84e81
environment:
- BACKEND_SERVICE=dashboard-backend
- BACKEND_PORT=3001
ports:
- “3000:80”
networks:
- dashboard
depends_on:
- dashboard-backend
restart: unless-stopped
healthcheck:
test: [“CMD”, “wget”, “–quiet”, “–tries=1”, “–spider”, “http://localhost/”]
interval: 30s
timeout: 10s
retries: 3
deploy:
resources:
limits:
cpus: ‘0.5’
memory: 256M
reservations:
cpus: ‘0.1’
memory: 64M
I’ve made these changes

  • changed env syntax
  • added maxmind env vars
  • set resource limits
  • make the maxmind database file available to the container (I shelled in to check it was there)

Its still acting in the exact same way. Runs for a couple of seconds with very high CPU usage (if no limit) and then restarts

you have to set a limit on resources if logs are large

EmilyxFox commented on GitHub:

you have to set a limit on resources if logs are 1.3 gb

My log file is no larger than 1.6MB