1. Credential clobbering on deploy
Config is not code — keep secrets out of source.
A zero-cost, self-hosted, hybrid-cloud AI system — engineered for reliability under hard constraints.
A self-hosted AI agent (Python + FastAPI, a hosted LLM on its free tier, a vector database for retrieval, and SQLite for state) that performs a scheduled daily task, keeps a retrievable knowledge base in sync, and serves a phone-friendly chat interface over HTTPS. The interesting part isn't the model call — it's the operations engineering around it: a hybrid local-primary / cloud-passive deployment with heartbeat-based failover, authenticated bi-directional sync, and fully idempotent infrastructure-as-code, all delivered inside always-free tiers without ever breaking the running service.
Home-lab VM runs the app under systemd and fires the daily task on a cron schedule. All real work happens here.
An always-free cloud VM sits idle, bound to localhost behind HTTPS, ready to take over only when needed.
Local stamps a presence heartbeat; cloud checks it and fails over only if local goes silent past a 25-hour window.
A token-gated endpoint exchanges knowledge vectors and history daily, merged last-write-wins, re-embedded on import.
shared sheet (heartbeat relay + logging)
▲ ▲
stamp heartbeat │ │ read heartbeat at 23:00 check
│ │
┌──────────────────┴──┐ ┌──┴───────────────────┐
│ LOCAL (PRIMARY) │ │ CLOUD (PASSIVE) │
│ home-lab VM │ │ always-free e2-micro │
│ • FastAPI (LAN) │ │ • app bound to │
│ • cron daily task │ │ 127.0.0.1 │
│ • heartbeat (local │ │ • Caddy -> HTTPS │
│ role only) │ │ • dynamic DNS │
└──────────┬───────────┘ │ • 22/80/443 only, │
│ │ SSH key-only │
│ token-gated └──────────┬────────────┘
│ HTTPS sync (last-write- │
└── wins, re-embed) ───────┘
remote user ──HTTPS──▶ Caddy ──▶ cloud app ──▶ chat UI + /audit
Heartbeat + 25-hour presence window guarantees exactly one daily run across two nodes — never a duplicate.
Token-gated daily exchange of knowledge vectors and history, last-write-wins, re-embedded on import.
A single playbook provisions the whole host — timezone, swap, firewall, runtime, app, TLS — re-runnable any time.
Read-only security audit scoring, gated service self-healing with a preview mode, and proactive TLS-expiry checks.
Auto certificates via reverse proxy, constant-time token auth, SSH key-only, root disabled, minimal open ports.
Every component lives inside an always-free tier, with a $1 budget tripwire that has never fired.
Reliability was earned the hard way. Each bug taught a generalizable lesson — full write-up in the post-mortem.
Config is not code — keep secrets out of source.
cron inherits nothing; load your environment explicitly.
“Works on my machine” has a version number attached.
Upsert arbiters have precise, engine-specific rules.
In sync, identity must be stable and origin-aware.
Keep slow, unbounded work off the request path.
A presence signal must be written only by its owner.
Retry with capped backoff for rare, high-stakes jobs.
Set the timezone explicitly; schedulers trust the clock blindly.