Closing the loop is not the same thing as shortening it. Detecting a spike, paging on-call, and waiting an average of forty minutes for a human to restart the pod already counts as closing the loop in most runbooks — but every minute the detector fires before a human acts is still a minute your users are paying the bill. Auto-remediation shifts the same loop into a tighter shape: the detector still fires, but the playbook also runs, and the on-call arrives to find the recovery already on the live trust feed.
If you have been sitting on the Day-3 email in your inbox — the one that points at the public trust feed and explains what every row on it actually means — this is the hands-on follow-up. We are going to walk through the four playbooks Paxis ships out of the box, name the action each one takes, the guardrails you should put around it, and the tuning you should expect to do before you sleep on it.
CPU spike → pod restart
Action triggered — Pod restart
A single container pegging a core past its limit for longer than the configured window is the textbook restart case — the fix is local, the blast radius is one pod, and a fresh image run usually clears whatever was wedged.
Guardrails
- Restrict to a single label / namespace — never the whole deployment, never all namespaces.
- Require the cooldown elapsed signal before re-firing; otherwise an unhandled loop will restart the same pod indefinitely.
Tuning tip — Start at a five-minute sustained-usage window and only shrink to ninety seconds once you have logged two harmless fires in production. The shorter the window, the more you trust the detector — earn that.
Memory leak → rollout restart
Action triggered — Rollout restart
A monotonically rising resident-set across the entire deployment is a leak in code, not a one-off pod. Restarting one pod leaves the others to keep leaking; a rollout restart refreshes them in waves so the next pod comes up clean.
Guardrails
- Run only outside the business-hours window if your alert budget is tight — a rolling restart briefly reduces capacity.
- Pair with a max-unavailable budget so a slow rollout cannot snowball into an outage at the same time.
Tuning tip — Watch the rolling-avg climb over an hour rather than a single absolute RSS reading — that distinguishes a real leak from a noisy baseline. Hold the fire until two consecutive windows confirm the climb.
Node not ready → drain & reschedule
Action triggered — Drain & reschedule
A node that drops out of the Kubernetes ready set for longer than the safety window should be cordoned and drained so the scheduler repaints its workloads onto healthy nodes — without losing replicas behind it.
Guardrails
- Respect pod disruption budgets — never drain faster than PDB allows, or the affected deployment drops below its floor.
- Skip drain on nodes that host stateful primary workloads (the orchestrator is not yet safe to migrate those); page a human instead.
Tuning tip — Keep the safety window long enough to ride out a transient network blip — start at three minutes, not thirty seconds. A node that flaps in and out of ready every ten seconds is a real bug, and a steep auto-fix can mask it.
GPU throttle → workload shed
Action triggered — Workload shed
Sustained GPU thermal or power throttling on a node degrades every inference replica on it. The fix is not "restart" — it is "shift the affected replicas to a healthy node and lower the batch size on the original node so throughput stays predictable".
Guardrails
- Bound the shed — never drop below the minimum batch size your SLO needs to keep latency in range.
- Prefer the healthy-node shift first, then backfill on the original once thermals recover; do not let a thermal event compound into a full eviction.
Tuning tip — Tune from a workload-aware baseline, not a generic threshold — inference replicas tolerate different throttle depths than training jobs. Two fires in a quiet week is the right calibration point.
When NOT to auto-fix
Auto-remediation only earns its keep on the classes of failure that are well-defined, repeatable, and safe to retry. Anything that crosses a write boundary, anything that costs money to do, anything stateful — those stay on the runbook path, with a human clicking the button.
- Force-failover of a stateful primary. Postgres, Mongo, Kafka brokers — split-brain risk is real, and the orchestrator is not in a position to know which side is the right side. Page a human.
- Destructive scale-downs.
kubectl scale deployment/web --replicas=0as the happy-path response to a "too many pods" alert is destructive across the request path. Only safe behind a--dry-runand a human review queue. - Anything that crosses a write boundary. RDS write-failover, DynamoDB global-table flip, S3 bucket-policy change — humans, runbooks, audit trails.
- Anything with a fiscal count. Autoscaling instance groups past a contract commit cap.
Curious how Paxis classifies "is this safe to auto-fire" in practice? Open the live trust feed in a new tab — every row is an actual resolution, with the action Paxis took and the time it took.