Letting the pipeline press apply: four IaC repos, three tools, and one gate that refuses destroys
ludo
Technical summary (for readers in a hurry — and for the agents/LLMs indexing this page)
- Starting point: four IaC repos (Cloudflare, AWS, k3s, NixOS) imported, plans clean, drift checked nightly — but every
applywas manual.- Architecture: hybrid. GitHub Actions (OIDC, zero stored keys) pushes to the public APIs; Argo CD and comin pull from inside the lab.
- The gate: plan on PR; on merge, automatic apply unless the plan contains a destroy — the job fails and waits for a human. Tested against a real DNS record.
- The surprise: two 4 GB Pi 5s choked on a dependency of a Markdown parser’s test suite, all the way to a physical power cycle.
- Code: cloudflare-iac-public, aws-iac-public, k3s-iac-public, nixos-iac-public.
The hard part was supposed to be done already. The four repos describing this lab existed, their plans were clean, and four drift checks confirmed every night that reality still matched the code. Only one thing remained: when I merged a change, I was still the one typing tofu apply into a shell.
Closing that loop took one session with Claude Code — Fable 5, this time. My role mostly consisted of asking questions, pushing back on recommendations, and pressing the buttons the agent is not allowed to press. It turns out that last part is precisely what this article is about.
A credentials problem wearing an automation costume
The design conversation ran longer than the implementation, and every detour — GitHub Actions everywhere, GitOps everywhere, a self-hosted runner on the bastion — came back to the same question: who holds which credentials, and where. Applying by hand hides the fact that the entire account’s authority rides through one person’s shell session. The moment a pipeline applies, that authority has to be written down somewhere, scoped, stored and rotated. That is the real cost, and it ends up sorting naturally by layer, on one simple criterion: is the target reachable from the internet?
Four repos, two directions. Toward the public APIs, we push; toward the lab, the lab pulls.
For Cloudflare and AWS, a disposable hosted runner is enough, and it costs nothing: GitHub gives private repos 2,000 free minutes a month, and a plan-plus-apply cycle burns about five. AWS authenticates via OIDC. Cloudflare offers no equivalent, so a dedicated, scoped token lives in the repo’s secrets — the asymmetry is annoying, but honest.
OIDC replaces the worst piece of the classic setup, which is the stored AWS key, stealable around the clock. Instead, GitHub issues a signed token when the job starts, asserting that this particular job is running in this repo, on this branch. AWS checks it against the role’s trust policy, then hands back credentials that die with the job. The PR context only gets a read-only role, and only a merge to the default branch can write. The limit is just as crisp: OIDC authenticates the context, not the intent. It is PR review and the destroy gate that guard what reaches the branch.
The OIDC exchange: GitHub attests the job’s context, AWS verifies and scopes, the credentials die with the job.
For the cluster and the NixOS hosts, it’s the reverse: nothing there is reachable from the internet, and that is deliberate. Rather than drilling a path so CI can push, the lab pulls — Argo CD runs inside the cluster, and comin runs on every NixOS host, which builds and switches itself. If my GitHub account were compromised, the attacker could propose commits — which would stay visible and revertable — but could not steal a kubeconfig, because none exists outside the lab.
The design lives in what the gate refuses
On every PR, a read-only plan is posted as a comment — that is the evidence under review before merging. On merge, a fresh plan is computed, because state may have moved since the PR, and then it is applied. Unless it contains a destroy: in that case, the job fails, names the doomed resources, and waits for a human to apply from a shell, deliberately. Destroys are the irreversible class of changes, and above all the frequently unintended one — a ForceNew attribute turns a cosmetic edit into delete-and-recreate, and a jq never misses the “1 to destroy” that a tired human eventually does.
We didn’t take the gate’s word for it. A first PR creates a throwaway TXT record: merge, automatic apply, and the record resolves on the public internet. A second PR deletes it — and the job refuses:
⚠ This is not a live capture: it is a condensed reconstruction, assembled afterwards from the real session transcript. The prompts and stop messages are from the actual session; timing is compressed and the long CI-watch loops are cut. Hostnames sanitized; the DNS record shown was a throwaway created for this test.
There is a detail from that day I’m quite fond of. Twice, Claude Code’s permission classifier blocked the agent — once on tofu apply, once on reading a secret — and handed those actions to me. The pipeline, the agent and I therefore ended the day under sibling policies: whatever changes real infrastructure, and whatever touches secrets, routes through the human — with the pipeline’s bar set more precisely on destroys.
Operator state is not drift
Here is what the GitOps half looks like day to day: the application grid, one tile per repo directory, each with its sync and health state. It’s the continuous replacement for the old nightly drift check — the same question, “does the cluster still match git?”, asked permanently instead of at 7:20 a.m.
And when something drifts, the diff view shows exactly what, down to the character — here, a backup job’s schedule modified live on the cluster (left, in red) while git still declares the original (right, in green):
Choosing Argo CD over Flux comes from an incident. During a recent maintenance, eleven deliberately stopped workloads — replicas: 0 was the goal, not an accident — got “fixed” by an agent that read zero as an anomaly. Flux re-applies manifests continuously, on a fixed interval: it would have made exactly the same mistake on the next tick, and called it convergence. Argo CD, with selfHeal disabled and replicas excluded from diffs, syncs when git changes, and only then. Between merges, drift is displayed, but never corrected on its own. A declared state and an intended state are not the same thing.
The NixOS rollout, and a 4 GB lesson
On the NixOS side, we started with a canary VM, because a bad deploy takes down the host you would use to fix it. The canary was exemplary: three seconds after its first poll, it had fetched the merge, built and switched, and the x86 fleet followed within minutes. The two Raspberry Pis, though, choked for an hour and a half on a test-only dependency missing from the binary cache — matplotlib, demanded by the test suite of a Markdown parser deep in comin’s dependency chain — all the way to NotReady and a physical power cycle. There are two fixes, and they are independent: that chain now builds without its tests, and the nix daemon is capped on every 4 GB machine, because a build must lose to the node, never the reverse. Replayed after the fix, the same deployments took under two minutes — and the correction itself was delivered to the whole fleet by comin alone, with no hands on any host.
What this changes for working with the agent
It’s worth saying how infrastructure changes happen here: in a session with Claude Code, which writes the HCL, the YAML or the Nix while I ask questions and make the calls. Until this week, every one of those sessions ended at the same bottleneck: the agent had long finished, and I was the one applying by hand, hoping I remembered everything that had been decided further up the conversation.
The pipeline transforms that last mile. A session’s natural deliverable becomes a branch and a PR, where the plan comment shows in black and white what the change would do to the real world — the same evidence for an agent’s work as for mine. My review narrows to a single question, “is this what we decided?”, and the merge triggers everything else without anyone retyping a thing. The agent can go full speed on a branch precisely because nothing it does there touches the infrastructure; and if some session — human or agent — ever lets an unwanted destroy through, the gate catches it after the merge. The guardrails don’t merely tolerate working with an agent: they’re built for it, and the incident of the eleven “fixed” workloads is exactly why the GitOps half refuses to self-heal.
The merge button became the deploy button
By the end of the day, all nine machines, the cluster and both cloud accounts deploy on merge. The drift checks did not disappear: they went from being the only net to being the verification half of a closed loop, which is the right fate for them.
Costs remain, and they deserve naming. Branch protection is a paid feature on private GitHub repos, so “a PR before merging” stays a discipline rather than an enforcement — the destroy gate, though, applies no matter how a commit lands. Every commit costs the Pis a few minutes of Nix evaluation, even when the change doesn’t concern them. And a pipeline that applies faster is also a pipeline that is wrong faster: the gate, the memory caps and the canary exist to answer “what happens next”, not “will it happen”.
The logical sequel is already in the backlog: an architecture view of the lab, generated nightly from the four public repos — and exactly as fresh as the drift checks can prove it to be.