Self-Healing Home Server
Always-on infrastructure agent with SSH access — diagnose issues, restart services, and recover from common failures across your homelab.
Automationadvanced~2h setup
- Tools
execmemorymemory_search- Channels
telegram- Uses
cronssh
Homelab and small-business infrastructure live in a gray zone — too much for "just SSH when something breaks", not enough to justify proper SRE. A self-healing agent watches the boring stuff (disk space, service health, backup completion), tries the obvious fixes itself (restart, prune, retry), and pings you only when human judgement is needed.
What it does
- Periodic health checks across configured servers (SSH-based)
- Auto-remediates known issues: full disk → run cleanup, dead service → restart, OOM → kill + log, expired cert → renew via certbot
- Escalates unknown failures to Telegram with full context
- Maintains a running incident log in memory
- Weekly digest: "this week I auto-recovered from N issues, escalated M"
What you'll need
- A list of servers to manage (with SSH key access from the Flowly host)
- Exec tool — built in
- Memory for incident log
- Cron for periodic checks
- An understanding of your stack so you know what "common" failure patterns look like
Setup
1. Set up SSH access
The Flowly host needs passwordless SSH to each managed server. Add
your ~/.ssh/id_ed25519.pub to each server's authorized_keys. Test
manually first: ssh user@server uptime.
2. Define the server inventory
Send to Flowly
Remember:
Servers to monitor:
- prod.example.com (web, postgres) — high priority
- backup.local (backup target) — medium priority
- pi-hole (DNS) — high priority
For each, store: name, ssh_user, ip, services, criticality.
Tag: "managed-servers".
3. Define remediation playbooks
Send to Flowly
Remember the safe-to-auto-remediate patterns. ALWAYS try in this exact order:
1. Disk > 90% full:
- Run "df -h" to confirm
- Run "journalctl --vacuum-time=7d" then re-check
- Run "docker system prune -af --volumes" if docker is on the host
- If still > 90%, escalate
2. Service is dead (systemd unit failed):
- Run "systemctl status <unit>" — capture last 20 log lines
- If error matches known transient (network, timeout), run "systemctl restart <unit>"
- Wait 30s, re-check status
- If still dead, escalate with full status output
3. SSL cert expires in < 7 days:
- Run "certbot renew --quiet"
- Verify new expiry
- Reload nginx/apache
NEVER auto-remediate:
- Database errors (data integrity risk)
- Authentication failures (could mask attack)
- Anything mentioning "manual intervention"
- Disk-full on a database volume
For those: escalate immediately.
Tag: "remediation-playbook".
4. Schedule health checks
Send to Flowly
Set up cron "infra-health" every 15 minutes:
For each server in managed-servers:
1. SSH in, run a health-check bundle:
- df -h
- systemctl --failed
- uptime / load
- free -m
- check critical services (HTTP 200 from nginx, postgres alive, etc)
2. For any anomaly, consult remediation-playbook
3. If a playbook applies, execute it; record outcome in memory tagged
"incident:<server>:<timestamp>"
4. If no playbook applies OR remediation failed, send Telegram alert:
"🚨 <server>: <issue summary>. I tried <attempted fix>, did not
resolve. Output: <details>. SSH in?"
Skip checks for servers that escalated < 30 minutes ago — give yourself
time to investigate before the next ping.
5. Weekly digest
Send to Flowly
Cron "infra-week" Sunday 8 PM:
Pull all "incident:*" from memory in the last 7 days. Group by server.
Send Telegram message:
📊 Infra week
prod.example.com — 3 incidents, all auto-recovered (disk × 2, service restart × 1)
backup.local — 1 escalation: backup target unreachable Tue 03:00
pi-hole — clean
Patterns: 4 auto-recoveries, 1 escalation.
Tips
- Start with read-only. First two weeks, the agent should only diagnose and report — no auto-actions. You build trust, you see what it would have done. Then enable remediation for confirmed-safe patterns.
- Audit log everything. Every SSH command + output goes to memory. When something breaks, you can reconstruct what the agent saw and did.
- Conservative playbooks. When in doubt, escalate. A false-alarm ping costs you 30 seconds; an auto-action that destroyed data costs you a day.
- Backup first, automate second. Make sure your snapshot/backup story is solid before granting any write access.
- Don't store credentials in the agent context. SSH keys live on disk, not in prompts. Database passwords live in the apps, not in what the agent reads.
- Phone-home checks are dangerous. Don't have the agent decide "let me reboot prod to fix this." Add an explicit allow-list of reboot-allowed hosts (if any).