How a Forgotten Postiz auto-setup.sh Left a Week-Old Zombie Process on My Homelab

Self-hosted Applications Security & Backup
, ,
1

How I Tracked Down a Week-Old Zombie Process on My Homelab Server (and What Actually Caused It)

The Symptom

One quiet Sunday afternoon the system status script spat out the classic warning:

=> There is 1 zombie process.

A single zombie is almost never urgent, but this one had been hanging around since 27 September. Time to dig in properly.

Step 1 – Confirm the Zombie Exists

ps aux | awk '$8 ~ /Z/'

Output:

homelab  1935019  0.0  0.0      0     0 ?        Z    Sep27   0:00 [auto-setup.sh] <defunct>

There it is: PID 1935019, owned by the homelab user, state Z, command name [auto-setup.sh] <defunct>.
The square brackets and <defunct> are the kernel’s way of saying “this process is already dead; only the process-table entry remains.”

A cleaner, more structured view:

ps -o pid,ppid,user,stat,start,cmd -p 1935019

    PID    PPID USER     STAT  STARTED CMD
1935019 1934909 homelab  Z      Sep 27 [auto-setup.sh] <defunct>

Step 2 – Find the Parent

The PPID column already told us the parent is 1934909. Let’s inspect it:

ps -o pid,ppid,user,stat,start,cmd -p 1934909

    PID    PPID USER     STAT  STARTED CMD
1934909 1934883 homelab  Ssl    Sep 27 temporal-server --env docker start

And confirm the binary:

ls -l /proc/1934909/exe

lrwxrwxrwx 1 homelab homelab 0 Sep 27 10:39 /proc/1934909/exe -> /usr/local/bin/temporal-server

Command line:

cat /proc/1934909/cmdline | tr '\0' ' '; echo

temporal-server --env docker start

So the long-running parent is Temporal Server (the workflow engine), started with --env docker.

What Actually Happened

At some point on 27 September an auto-setup.sh script was executed under the homelab user. That script finished (or was killed), but its parent — the Temporal Server process — never called wait() / waitpid() to collect the child’s exit status.

The result is a classic zombie: the process is gone, its memory and file descriptors are released, but the process-table slot remains until the parent reaps it.

(Note: official Discourse does not ship an auto-setup.sh. The script came from a different setup tool — in this case a Postfix-related installation helper that the Temporal environment had pulled in or that had been run manually.)

Why Zombies Are (Usually) Harmless

A zombie consumes:

  • one process-table entry
  • a few bytes of kernel memory for the exit status

It uses zero CPU and zero user-space memory. A single zombie that has been sitting there for a week is therefore almost never a problem. You only start caring when:

  • you have dozens or hundreds of them, or
  • the process table is approaching pid_max.

Remedies

Option 1 – The polite way (preferred)

Send SIGCHLD to the parent so it is forced to reap its children:

kill -CHLD 1934909

Then check again:

ps aux | awk '$8 ~ /Z/'

If the zombie is gone, you’re done.

Option 2 – Restart the parent cleanly

Because the parent is a long-running service (temporal-server), the cleanest operational fix is to restart it the normal way (systemd, docker compose, whatever you use to manage it). When the parent exits, the zombie is re-parented to PID 1 (systemd), which immediately reaps it.

# example – adjust to your actual unit / compose service
systemctl restart temporal
# or
docker compose restart temporal-server

Option 3 – Nuclear option (only if the parent is stuck)

kill 1934909          # SIGTERM
# wait a few seconds
kill -9 1934909       # only if it refuses to die

After the parent dies, the zombie vanishes within a second or two.

Option 4 – Just leave it

If the system is otherwise healthy and you don’t want to touch Temporal right now, you can safely ignore a single long-lived zombie. It will disappear the next time the parent is restarted for any other reason.

Quick Diagnostic Cheat-Sheet

# List all zombies
ps aux | awk '$8 ~ /Z/'

# Structured view of a specific zombie
ps -o pid,ppid,user,stat,start,cmd -p <zombie-pid>

# Inspect the parent
ps -o pid,ppid,user,stat,start,cmd -p <ppid>
ls -l /proc/<ppid>/exe
cat /proc/<ppid>/cmdline | tr '\0' ' '; echo

# Force the parent to reap
kill -CHLD <ppid>

Take-aways

  1. One zombie is almost never an emergency.
  2. Always look at the parent, never try to kill the zombie itself.
  3. Long-running services (Temporal, databases, message brokers, etc.) are the most common culprits for leftover zombies because they often forget to handle SIGCHLD for short-lived helper scripts.
  4. Keep a note of any one-off setup scripts (auto-setup.sh, install helpers, etc.) that run under long-lived processes — they are frequent sources of this exact problem.

That’s the full post-mortem of a week-old [auto-setup.sh] <defunct> on a Temporal-powered homelab. lol