swamp-ops

Laptop-side operator automation for the seanfarm platform, run with swamp. This repo holds the typed models, workflow DAGs, and reports for break-glass and verification ops that do not belong in the cluster's delivery path.

Boundary: the pipelines control plane and Argo own everything in-cluster. Swamp owns only laptop-side operator procedures — the ops that used to live in transcripts and deleted scripts.

Ops

Workflow	Purpose
fleet-mirror-check-v2	The mirror-verification runbook: git ls-remote compare of forgejo vs codeberg for every fleet pair. Guarded forEach — re-runs only re-check undelivered repos; --input force=true for fresh evidence.
offsite-health	The offsite-backup probe: forced fleet mirror check + codeberg packages-quota headroom. With the backup crons retired (2026-08-17) this is the only proof the only backup mechanism delivers. SCHEDULED: launchd farm.sean.offsite-health runs it AND ops-sweep 08:23/20:23 local (scripts/install-offsite-health-launchd.sh; silent green, macOS notification on red; log ~/Library/Logs/swamp-offsite-health.log).
ops-sweep	The operational truth sweep: image refs resolve, board clean, cron census, worker kernel convergence, ArgoCD Synced-vs-operation honesty.
rebuild-preflight	The make-rebuild go/no-go: nested offsite-health + cnpg-backup-gate (backupMaxAgeHours, default 6; pass 1 on the final pre-destroy run), pins-vs-fleet, storage plane, board, cron census — plus allowed-failure warnings for LE-prod and the pipelines-backup snapshot restore.
kernel-roll-preflight	Run before declaring a kernel deploy: storage plane, board, sean/linux mirror delivered (verify-source-backup's dependency), workers converged.
kernel-roll-postflight	Roll acceptance: whole fleet (master included) on --input expectTag=<generation>, board clean, ArgoCD honest.
cnpg-backup-gate	The off-cluster backup runbook conditions: ContinuousArchiving, LastBackupSucceeded, backup age under maxAgeHours (input, default 12; use 1 before a rebuild).
gated-fleet-recheck	Approval-gate drill: a human swamp workflow approve in front of a nested fleet check.

Extensions (local, no registry)

• extensions/models/mirrorpair.ts — @seanfarm/mirrorpair: delivery verdict per branch from git ls-remote on both remotes.
• extensions/models/codebergquota.ts — @seanfarm/codebergquota: packages-quota headroom with a failure floor (cred read at runtime from the cluster secret; nothing stored).
• extensions/models/argocdhonesty.ts — @seanfarm/argocdhonesty: Synced claims vs last-operation evidence (the webhook-denied-apply class). op.revision != sync.revision is deliberately NOT a signature.
• extensions/models/mayastorhealth.ts — @seanfarm/mayastorhealth: DiskPools online + stateful PVCs bound (volume-level health still needs the kubectl-mayastor plugin).
• extensions/models/rebuildpins.ts — @seanfarm/rebuildpins: rebuild pins on FORGEJO MAIN vs the live fleet kernel, plus the ACME issuer state.
• extensions/reports/mirror_delivery.ts — @seanfarm/mirror-delivery: per-branch table after every check.

Standing conditions (adoption terms, 2026-08-13)

1. Pinned source build only. ~/src/swamp at tag v20260813.161427.0-sha.03547878, built with deno task compile. Never swamp update, never curl|sh. Move the pin deliberately.
2. Telemetry and update checks OFF — enforced by the /opt/homebrew/bin/swamp wrapper (SWAMP_NO_TELEMETRY=1, SWAMP_NO_UPDATE_CHECK=1). Telemetry is ON by default upstream.
3. No registry extensions. Local extensions/ only. Anything pulled from the registry is arbitrary unsigned TypeScript and needs review first.
4. No real secrets in swamp vaults until an OpenBao vault-provider extension exists. local_encryption with auto_generate keeps the key beside the ciphertext — fine for the dummy trial secret, not for credentials. Cluster access uses the ambient repo kubeconfig.

Gotchas (hard-won)

• Data name latest is reserved; dependsOn entries are objects.
• A guard that throws FAILS the step. data.latest() on an absent record is null and .attributes throws — null-check first, && short-circuits (swamp CEL has no ??).
• CLI data get shows the payload under .content; CEL sees .attributes.
• A FAILED command/shell run records no result data — failure evidence is in run history, so asserts on latest data see the previous run.
• Report context modelType is an object, not a string. Custom reports must be attached via reports.require on definitions.
• Deleting an auto-created model leaves a stale file in .swamp/auto-definitions/ that keeps the name claimed (upstream bug); remove the file to repair.