Reliability and security: encrypted backup capsules and locked-down secrets
Encrypted nightly backups copied offsite to cloud storage and a private GitHub repository, locked-down secrets, and a scripted rebuild on a fresh server.
- Timers
- hermes-backup · hermes-offsite · hermes-offsite-verify · hermes-capsule-drift
- Offsite Target
- Encrypted offsite copies (cloud storage plus a private GitHub mirror)
- Secrets Isolation
- Locked-down secret files · a password vault that only fills the matching website
Problem
Agents that can run commands on a server can leak keys, slowly change settings by accident, or lose work when the server restarts. Reliable operations require continuous drift audits, automated offsite disaster recovery, and strict secrets isolation.
What I Built
- Backup schedule: four systemd timers run a daily backup, a daily offsite copy, a weekly check that the offsite copy still opens, and an hourly check for changes; a separate hourly job snapshots agent memory.
- Encrypted backups: each backup is encrypted with AES-256 before it leaves the server, then copied to Backblaze B2 cloud storage and mirrored to a private GitHub repository.
- Disaster Recovery Runbook: Scripted restore procedure capable of bootstrapping the full multi-agent environment from bare Ubuntu 24.04.
- Secrets Handling: Sensitive API tokens and credentials live in locked-down environment files and origin-bound vaults. Agents are set up so credentials are filled by the vault and never passed in conversation.
Egress and anonymity
Web research that needs anonymity goes out through rotating egress over Tor and Cloudflare WARP, so no single route carries it and a failing route is dropped automatically.
- Load balancer: a small proxy (glider) spreads requests across five Tor routes, and a second pool prefers Cloudflare WARP and falls back to Tor when WARP is slow or down.
- Health checks: Glider checks every route every 30 seconds and drops a route after 2 failed checks, so traffic keeps flowing over the routes that still work.
- Scope: This covers only the research and scraping traffic that is routed through it. It changes which route a request takes, nothing else.
- Encrypted offsite backups: backups are encrypted, copied to cloud storage and a private GitHub repository, and a weekly job downloads one and checks it still opens.
Outcome
A written restore script, and a weekly check that the latest offsite backup downloads and opens. Hourly checks for unexpected setting changes.
Stack
systemd · OpenSSL · rclone · Backblaze B2 · GitHub · Bash · Python · Linux