binjovi

Binjovi is a portable Elixir and OTP CI/CD control plane. Kubernetes is its first deployment target. Kubernetes is not its durable state model or its only execution engine.

The service resource foundation stores service intent, ownership, queued effects, and verification in PostgreSQL. The Services workspace, API, and CLI manage approved Forgejo policies, Hydra clients, and Kratos identity metadata. The migration inventory records the remaining operator responsibilities and the next provider work.

The first pass uses sean/bytey to measure the minimum control-plane cost. Bytey has almost no product work, so it exposes coordination overhead.

Current slice

The current slice contains:

• pure Build, Release, and Deployment models with immutable source identity;
• a typed, shell-free Bytey execution plan;
• a PostgreSQL authority with automatic migrations, provider-delivery deduplication, source deduplication, and compare-and-set transitions;
• one short-lived :gen_statem coordinator for each active Build;
• pod-scoped lease owners and fencing epochs for safe controller takeover;
• signed Forgejo webhook wakeups for Bytey and for Binjovi, with a bounded five-minute recovery scan for missed deliveries;
• a protected external plan repository with exact-revision reads, full catalog validation in the control-plane process, atomic PostgreSQL activation, and an embedded bootstrap catalog;
• canonical Forgejo pull-request Changes with frozen plan digests and the binjovi/ci commit status;
• restricted Argo Workflows that run Bytey and the Go library projects through one rootless, daemonless BuildKit process for each Build;
• durable status delivery, pull-request head supersession, Workflow termination, archived logs, and live log folds;
• a gitops-flux taxonomy that runs provider-guarded seanfarm and platform-seanfarm with an AlmaLinux 10 validator and typed Flux Deploy targets;
• a gitops-argocd taxonomy that validates Argo CD source repositories and observes an exact released revision across a declared Application set, with Ory and Olly provider-guarded;
• an in-memory Bytey floor probe that measures the control contract without creating project Build records;
• authenticated, idempotent release commands with durable stage, lease, fencing, retry, and external-effect evidence;
• frozen Build batches, ordered trunk integration, one separate combined trunk Build execution, main promotion, stable patch tags, Forgejo Releases, and immutable artifact identity;
• multi-item Dockerfile-set Builds that upload digest-qualified OCI candidates directly to the in-cluster registry while durable evidence uses the public quarantine reference, followed by a separate trusted Release publisher that verifies the same digest at each final repository;
• a go-operator-image taxonomy that runs the required Go tests, builds one Linux AMD64 and ARM64 image index for each exact revision, and records one digest-qualified OCI artifact;
• separate candidate and Release publisher identities for Go operator images; project code can write only to its quarantine path, and the trusted Release publisher can write only to the four final operator repositories;
• a server-package taxonomy with a gate-only pull-request plan and one full combined trunk plan that builds each candidate DEB and RPM, creates a signed private APT and DNF repository, and installs through that repository;
• separate package staging, read-only fetch, candidate finalization, signed public repository publication, public verification, and Forgejo Release asset identities; repository content and each member have immutable SHA-256 evidence;
• a Builder Images shadow policy that can create candidate Build evidence but cannot release, publish, or deploy until a separate authority change;
• independent deploy commands with exact-revision target verification;
• revision Releases for GitOps repositories, with a v<12-hex> Forgejo tag, a pinned AlmaLinux 10 deployer, exact Flux readback, and automatic rollback;
• an operator dashboard with only Builds, Releases, and Deploys; Build folds contain PR Change evidence, execution attempts, and logs; each Release fold contains its exact ordered Build batch and combined trunk Build evidence;
• an OTP release and AlmaLinux 10 container image;
• one revision-stamped binjovictl escript in each image, with public immutable bytes and checksum, a private version API, readiness validation, and a fail-closed atomic update command;
• separate long-lived native agents for Binjovi admission Builds and Release image Builds, with independent identities, sharing ONE BuildKit daemon and cache (the per-lane daemons were merged on 2026-09-07: every committer is authenticated, so a Change carries the trust of the trunk it targets, and five caches held largely the same layers);
• immutable source revisions, complete image patches, and exact-digest Argo CD delivery;
• an isolated, read-only staging instance, a 15-second availability gate before automatic production promotion, and a separate production gate;
• three control-plane replicas on separate nodes, with a two-copy disruption floor during planned node rolls.

The Bytey floor probe stays in memory and does not create project activity. Bytey and Binjovi pull requests run the real CI path. An authenticated user or the narrowly scoped release CLI can freeze an ordered list of successful Builds. Binjovi fails the command if any Change head or base became stale. It then tries a fast-forward-only merge for each selected pull request in order. If an earlier train car advanced trunk, Binjovi uses the repository's permitted rebase merge for the next car. It does not run CI between items. After the last integration, Binjovi runs the project plan in a new Build for the exact combined trunk revision. This is a new execution even when one selected PR has the same revision. Only a successful combined Build can move main, create the next stable patch tag, and record the Forgejo Release. A separate deploy command updates the bytey-floorplan ConfigMap with resource-version compare-and-set. Binjovi records a Deploy only after exact readback.

GitOps targets can add a trusted availability policy after exact convergence. The policy is part of the Binjovi release, not project content. Each endpoint URL addresses its Kubernetes Service port, not its container port. The deployer can also prove registry writes by committing and reading a random OCI blob during preflight. Trusted code supplies the credential file paths and keeps this check out of the sustained window. The deployer requires five consecutive preflight passes and then a zero-failure 60-second window. The preflight phase also has a 60-second limit, so a failed endpoint produces a summary instead of waiting for the Deploy deadline. Every HTTP request has a strict five-second limit. A request that takes more than five seconds makes the target unavailable. Binjovi stores the summary and a bounded copy of the terminal availability Workflow log with the Deploy. Flux targets can use the same evidence to roll back; Argo CD targets fail closed and require a new recovery Release. The Binjovi Workflow has a separate 90-second deadline. This includes preflight, the five-second observation window, and terminal summary and log publication. It prevents the Workflow deadline from racing a valid gate result.

Binjovi pull requests use the admission native agent. External plan version 12 uses the Release native agent for the combined image Build. Shipwright remains the version 11 fallback during the controlled change. The first new release is v0.1.0; later releases increase the stable patch version. Each Release records the exact OCI digest. A separate deploy command gives Argo CD the immutable source commit and all image patches in one guarded Application update. It creates no deployment branch or deployment commit. Argo CD applies the read-only instance at binjovi.dev. The native availability agent then verifies the Service, dashboard activity, public IPv4 reachability, public A and AAAA records, and the protected UI. The production gate also verifies exact native execution readiness. Before it starts that gate, the rollout observer requires the control plane, admission agent, and Release agent Deployments to be fully ready at the exact revision and image digest. Its two cross-namespace grants can read only the named agent Deployments. Forgejo, Argo CD, and registry health belong to their own project Deploy gates. A fault in one of these services cannot fail a Binjovi functional gate. The gate needs one successful preflight round and then zero failures for five seconds, with at least 15 samples. DNS, UI authentication, dashboard rendering, and agent health run only in preflight. Internal and public readiness remain sustained. The probe writes the exact summary to the Workflow output and logs. Binjovi stores the summary with the Deploy command even when Argo marks the Workflow as failed. The API, CLI, and dashboard show the sample count, failure count, and failed check names. A retry clears the old summary before it starts a new attempt. Only a successful gate lets policy copy the same Release to deploy for production. A distinct production Workflow repeats the checks against the production Service and binjovi.sean.farm. Production succeeds only after all three replicas report the exact source revision and image digest and the production availability gate passes. A production gate failure records terminal evidence. It does not roll back or hide the deployed state. The server does not retry a failed production gate for the same Release and Target pair. A later production attempt must come from a new Release. This keeps the failed decision terminal and keeps both attempts in the audit trail. A Deploy that stays pending for 15 minutes fails with deployment_timeout. This bound includes a rollout that Kubernetes cannot start. The operator can retry the staging Deploy after the platform fault is fixed. A failed staging Deploy has an authenticated retry action in its dashboard fold. Each retry carries a new durable attempt number. Flux and Argo CD include that number in the trusted Workflow name and identity annotations, so a completed Workflow cannot satisfy a later retry. Binjovi resets its deterministic probe before its new attempt. The API rejects direct production retries.

If an availability Workflow terminates before it writes a summary, Binjovi stores its bounded controller failure details instead: Workflow phase, message, and failed node messages. binjovictl deploy DEPLOYMENT_ID prints this evidence. This keeps failure diagnosis in the Binjovi API and CLI; it does not require direct Kubernetes access.

Each released revision records its complete accepted-to-verified duration, with separate Build, Release, and Deploy timings in the dashboard.

Run

Install Elixir 1.20 with Erlang/OTP 29, then run:

mix deps.get
make fast
make build-check
make integration-check
make check
make floor
make cli

All Make compilation and test commands treat compiler warnings as errors. Expected runtime failure logs are separate from compiler warnings.

make build-check is the pull-request Build gate. It runs compilation, static checks, and unit tests without PostgreSQL. make integration-check runs only the PostgreSQL integration suite. make check runs both in that order. The phase boundary is strict: Builds compile and run unit tests, Releases run integration tests for the exact combined revision, and Deploys run functional availability checks against the deployed artifact. Staging also requires two browser wiring checks for the exact release source. It runs on the admission agents through BuildKit alongside rollout checks. Its result is stored under browser in deployment verification; a failed check blocks production promotion. Each Deploy retry uses a new task. Browser checks are outside PR admission. make check-local still includes them in its full container check. Without a host PostgreSQL, make check-local runs the complete gate inside a container with docker or podman. The first run compiles the dependencies; the engine's layer cache makes every later run use the changed layers only.

The PostgreSQL integration suite truncates its database between cases. If BINJOVI_TEST_DATABASE_URL is set, its database name must end in _test. Both the shell helper and the connected test process enforce this rule before they run migrations or destructive SQL.

Start the HTTP service:

mix run --no-halt
open http://127.0.0.1:4000/
curl http://127.0.0.1:4000/health/ready
curl http://127.0.0.1:4000/api/v1/floor
curl http://127.0.0.1:4000/api/v1/projects/bytey/activity
curl http://127.0.0.1:4000/api/v1/projects/binjovi/activity
curl http://127.0.0.1:4000/api/v1/projects/binjovi/authority
curl http://127.0.0.1:4000/api/v1/projects/binjovi/targets
curl http://127.0.0.1:4000/api/v1/projects/binjovi/availability

The Kubernetes dashboard is https://binjovi.sean.farm. Select bytey to see real pull-request Builds with folded Forgejo Change evidence, Releases, and marker Deploys. Select binjovi to see each image Build, OCI Release, and Argo CD Deploy. The dashboard refreshes durable PostgreSQL activity every five seconds.

The HTTP listener reports rejected protocol requests as a bounded category, such as HTTP request rejected (request_line). It does not log the rejected bytes or Bandit's raw protocol exception. Those bytes can contain credentials before the router receives a request. Use the time, Pod, and gateway request ID to investigate a rejection. Do not enable raw protocol logging on a live listener. Request handling and HTTP status codes are unchanged.

Body readers must return the updated Plug.Conn on success and failure. Bind that connection before further validation: an Elixir with does not pass its new variables to else. An old connection can make Bandit read the consumed body again and take bytes from the next request on a reused connection. The real TCP tests in http_body_state_test.exs split headers from body bytes and then send a second request. Keep these tests in the cluster test suite.

Read durable status and provider authority, list the exact release candidates, or release the complete current batch:

apps/binjovi_cli/binjovictl status platform-seanfarm
apps/binjovi_cli/binjovictl plan-source
apps/binjovi_cli/binjovictl service get forgejo/seanfarm/bytey/branches/trunk
apps/binjovi_cli/binjovictl attempts BUILD_ID
apps/binjovi_cli/binjovictl attempt-log ATTEMPT_ID main
apps/binjovi_cli/binjovictl availability-log DEPLOYMENT_ID
apps/binjovi_cli/binjovictl authority platform-seanfarm
apps/binjovi_cli/binjovictl targets binjovi
apps/binjovi_cli/binjovictl deploy DEPLOYMENT_ID
apps/binjovi_cli/binjovictl retry-deploy DEPLOYMENT_ID
apps/binjovi_cli/binjovictl availability binjovi
apps/binjovi_cli/binjovictl builds bytey
apps/binjovi_cli/binjovictl authority tetragon
apps/binjovi_cli/binjovictl retire-pipeline-bot tetragon
apps/binjovi_cli/binjovictl release bytey --all-ready --wait
apps/binjovi_cli/binjovictl version
apps/binjovi_cli/binjovictl update

These read commands use the Binjovi API and durable PostgreSQL evidence. They do not call Kubernetes. The authority command asks the running control plane to apply its configured Forgejo provider guard. The availability command does not treat a successful rollout as service availability. It reports not_checked unless the Deploy has recorded probe samples. For a project with more than one configured target, every target must have successful probe evidence before the overall state becomes available. attempts reports the executor reference, failure reason, archived log containers, and bounded active Argo node messages. A waiting Build therefore shows its scheduling or image-start reason without direct Kubernetes access. The response omits Workflow inputs, arguments, environment values, and Secret references. attempt-log reads one archived Build container log through the same authenticated API. availability-log reads the bounded, durable log from the availability Workflow for one exact Deploy. It falls back to the retained Argo Pod only for Deploys that completed before Binjovi stored the log. The fallback selects the Pod by its exact Workflow label and reads its workload container (main for an Argo container template). It cannot select a Pod from another Workflow.

retry-deploy uses the authenticated Binjovi API. It retries only a failed staging Deploy that the server policy permits. The server rejects production retries and a second staging retry. A permitted retry creates new trusted Workflow evidence for its numbered attempt; it never reuses a completed result.

init-trunk PROJECT --from-main is a one-time shadow-project adoption command. It reads the exact Forgejo main revision through the Binjovi API. The server creates trunk only when it is absent. A changed main, an existing divergent trunk, or a project that is not in shadow mode makes the command fail. The command does not use kubectl or a Forgejo token on the operator computer. Tetragon used this command during adoption and is now provider-guarded.

retire-pipeline-bot PROJECT removes only the named legacy collaborator CR after the running provider guard proves that Binjovi owns main, trunk, and tag authority. Kubernetes RBAC must also name that exact CR. A missing CR is an idempotent result only after a direct Forgejo read with a separate read-only token confirms that pipeline-bot is absent. The command cannot delete repositories, push mirrors, protections, or arbitrary Kubernetes resources.

--wait has a 30-minute default deadline. Use --wait-timeout SECONDS to change it. The wait loop retries transient transport errors, rate limits, temporary gateway errors, and malformed temporary responses. It does not repeat the release command. Authentication failures and terminal Release failures or cancellations stop immediately.

By default, the CLI reads ~/.config/binjovi/client-secret. It does not call kubectl and does not use the macOS keychain. A portable installation can set BINJOVI_CLIENT_ID and either BINJOVI_CLIENT_SECRET_FILE. Keep the file private (chmod 600); the CLI refuses files with group or other permissions. BINJOVI_CLIENT_SECRET remains available for compatibility, but use a secret file for normal operation. Hydra grants this client only the binjovi.release scope. Automatic releases stay disabled.

Each Binjovi image builds one CLI artifact from the same exact Git revision as the server. Install it from https://binjovi.sean.farm/downloads/binjovictl. The adjacent .sha256 route supplies its checksum. binjovictl update reads the private server version with its normal short-lived identity, downloads the public artifact and checksum, verifies the size, digest, API major, and exact revision, then replaces the current executable with one same-directory rename. Any failed check leaves the installed executable unchanged. Every remote CLI command checks the server API major before it performs the requested operation.

The two go-library-tag projects use provider-guarded release control. Binjovi checks the live Forgejo main, trunk, and tag protections before each release effect. A catalog change alone cannot transfer repository authority.

The four go-operator-image projects start in shadow mode. A pull-request Build runs go test, go vet, the repository test suite, and controller runtime envtest. It then builds one OCI image index for linux/amd64 and linux/arm64. The untrusted Build can write only to the project quarantine path. The combined trunk Build repeats the same tests and build after Binjovi integrates the selected pull requests. A trusted Release Workflow copies only that exact digest to the project final repository, signs it, reads it back, and verifies the signature with an independently exported public key.

Shadow mode permits Build evidence only. It does not permit a Release. Each operator moves to provider-guarded control in a separate change after real shadow Builds prove the new path.

Forgejo calls /hooks/forgejo through the cluster-local Service. The endpoint verifies the payload HMAC before it wakes a reconciler. The payload is only a hint. The reconciler reads canonical source state from Forgejo and durable phase state from PostgreSQL. A five-minute scan repairs a lost webhook. Normal source discovery therefore takes one signed delivery round trip, not a fixed polling interval. Configure the Forgejo hook with the pull_request event. Push events do not create Builds or Releases.

Set the pull-request base to trunk. Each project repository keeps main as its Forgejo default branch, so the base is not correct by default. A pull request with another base makes no Change and no Build, and the only symptom is a wrong_base_ref warning in the log.

Each phase has a durable wall-clock boundary. Build time is acceptance through terminal Build state. Release time is release-command acceptance through immutable artifact observation. Deploy time is deploy-command acceptance through independent target verification. The accepted-to-verified total runs from Build acceptance through Deploy verification. It can include policy wait time between the independent phases. A controller restart does not reset a phase timer. A pending Deploy cannot report Running without a bound. Its terminal deadline is 15 minutes from command acceptance.

An independent watchdog checks active Builds, Release commands, Deploy commands, and native tasks each second. It uses the 20 most recent comparable successful actions. After five samples, the adaptive limit is the greater of the mean plus two sample standard deviations and the mean plus five seconds. The normal hard timeout is always the upper limit. The first stall fences the current owner and starts one recovery with the same frozen input. A second stall fails the action with adaptive_stall_limit_exceeded. PostgreSQL stores the samples, limit, elapsed time, last durable effect, and recovery result. Lease renewal does not reset the action clock and cannot hide a blocked call.

Binjovi Builds and Releases share one persistent BuildKit daemon and its content-addressed cache. Separate native agents keep their OAuth identities, worker labels, and mutual TLS clients. Only the Release lane receives the registry push credential. Cache reuse depends on matching build inputs; a different source snapshot must pass its own required gates. Build logs stream while containers run. At terminal state, Binjovi also captures any container log that does not yet have a durable MinIO chunk. The MinIO bucket rule removes each log object after 30 days. An hourly worker then deletes each log chunk row after 31 days, so a row always outlives its object. A read of an expired log answers 410 logs_expired. It does not answer 503, because a retry cannot bring the object back. One expired chunk does not hide the chunks that remain. The Build fold selects the failed command, or the primary build command, by default. A successful executor does not become a successful Build until its required command log is durable. Binjovi keeps the Build active and retries the same executor evidence for five minutes. The dashboard shows this derived phase as Archiving logs. The Build fails with required_log_evidence_timeout only when that deadline expires. The Deploy fold shows the observed publish, rollout, availability, and completion evidence as one durable transcript. It also shows the bounded, durable availability Workflow log. binjovictl deploy DEPLOYMENT_ID also prints structured evidence for a failed gate, including its sample and failure counts. A replacement controller adopts the same deterministic Workflow or BuildRun after a lease expires. Fence epochs stop an expired controller from changing the recovered run. Pull-request Builds run make build-check with MIX_ENV=test. Production compilation and packaging run during Release, outside the admission dependency graph. PostgreSQL integration tests also run during Release for the exact combined revision. Release trusts the frozen pull-request results for admission. BuildKit reuses matching dependency, test-compilation, and production layers from prior solves. Server and agent packages do not embed the Git revision; only the CLI is recompiled with the exact revision stamp. Final image assembly requires the integration marker and stamped CLI. Staging and production use that same final image digest; deployment does not rebuild it.

Floor scope

mix binjovi.floor and /api/v1/floor measure this path:

validated Forgejo-shaped event
  -> atomic delivery and Build acceptance
  -> OTP coordinator start
  -> immutable Bytey plan creation
  -> fake executor start and completion events
  -> terminal Build state

The floor measurement does not include Forgejo synchronization, PostgreSQL, worker network time, repository checkout, BuildKit, or Bytey compilation. It remains a stable lower bound while the dashboard shows the durable production observation path.

See docs/architecture.md for the design boundaries and docs/pipeline-lifecycles.md for the three-section project model and the Build and Release state machines. See docs/native-execution.md for the server and agent boundary and the Argo migration gates. See docs/migration-roadmap.md for the current Pipelines migration gates, project ledger, and performance targets.

Native node-image tasks

The node_image_job.v1 task keeps an admission-agent lease while Binjovi controls a separate Kubernetes Job. Only the bake container is privileged. The Job uses a file-backed disk and cannot provision a node. The agent gets no Kubernetes credential. Job control and cancellation use the control-plane client with its configured namespace, token path, and CA path. Each control request must match the current lease and fence. Completion requires the exact candidate proof from a Pod owned by that Job.

The task timeline records the checkout, bake, build, and collect container times. The task log contains bounded container log tails. Container times come from Kubernetes timestamps; they do not provide syscall resolution. CPU, off-CPU, and syscall profiles inside the bake VM are not collected by this task. The existing legacy Job plans remain readable.