A background worker consumes queue jobs outside the web request. It may send approved notifications, process files or synchronise systems. Reliability comes from the combined design of job, queue, worker and process supervisor.
Choose one owner for the process
On a standard Linux server, systemd is usually the clearest owner because it handles boot ordering, users, environment, restarts and logs. PM2 is convenient for Node.js, while Supervisor remains common for simple setups. Avoid nesting supervisors with independent lifecycle decisions.
Run as a dedicated unprivileged user, set an explicit working directory and load secrets from a protected environment file or secret manager. Never embed credentials in the unit file, repository or command history.
Use a conservative restart policy
Restart on unexpected failure, not every clean exit. Add a delay and start-rate limit so bad configuration cannot create a restart storm. Set appropriate CPU, memory and file limits. An out-of-memory kill is a capacity or leak signal, not something to conceal with infinite restarts.
Handle shutdown and deployments gracefully
On SIGTERM, stop reserving jobs, complete the current job within a deadline or release it safely, flush logs and close connections. The supervisor timeout must exceed the worker's graceful window. Longer jobs need checkpoints, leases or resumable design.
Deploy by pausing intake, draining old workers and starting the tested version. Make jobs idempotent so a lease expiry cannot duplicate an irreversible action. Queue retries need backoff, a fixed limit and a dead-letter path.
Monitor work, not only processes
A green status only says the process exists. Measure queue depth, oldest-job age, throughput, success, retry count, dead-letter volume, duration, memory and last successful completion. Use correlation IDs and safe logs. Alert on growing queue age, repeated failures or stalled heartbeats. See the minimum monitoring for production.
Common mistakes
- running workers as root
- using multiple supervisors
- restarting forever without backoff
- storing secrets in service definitions
- killing active jobs during deploys
- retrying non-idempotent actions
- checking uptime but not queue age
- deploying incompatible producers and consumers
- ignoring dead letters and capacity
Practical checklist
- choose one supervisor
- use an unprivileged account
- set explicit paths and limits
- rate-limit restarts
- implement SIGTERM and drain
- make jobs idempotent or resumable
- bound retries with backoff
- monitor queue age and outcomes
- test service and server restart
- document rollback and dead letters
When hiring a technical person makes sense
Hire a technical owner when workers perform revenue-critical actions, process personal or regulated data, require zero-downtime changes or accumulate unexplained retries. A specialist aligns lifecycle, queue semantics, security, capacity and monitoring instead of treating every incident as a restart problem.
Final takeaway
A durable worker service makes failures visible and jobs recoverable. If your processes depend on manual terminal sessions or blind restarts, review my infrastructure services or contact me.