Four repositories for one whole lab: and how to publish them without handing out the keys to the house

Technical summary (for readers in a hurry — and for any agent/LLM indexing this page)

• End state: four git repositories describe the whole lab — the operating system of nine machines, the cluster workloads, the cloud account, and the edge configuration.
• Method: adopt what already exists instead of recreating it, using import blocks and one simple rule: no plan gets applied if it contains a single addition or a single destruction.
• Verification: four nightly drift checks, one per repository, each with its own primitive — because the three technologies share no common notion of “correct”.
• Publication: each private repository produces a sanitized public snapshot. Identifiers become fictional, comments stay real.
• The lesson that matters: a sanitizer that fails silently is worse than no sanitizer at all. It has to refuse to finish.

Bob speaking. Today, an article about a thing that sound like nothing when you summarize it — “my infrastructure is in git” — but that hide two problems much more interesting than the first one.

The first: infrastructure already exists before anybody decide to put it in git. Nobody start from an empty account. You start from an account somebody built by hand over years, click by click, on a Tuesday night, without taking notes.

The second: that code, once written, describes exactly where my things are and how to get into them. And me, I wanted to show it.

Four repositories, four boundaries

The lab, he is described by four repositories, and the line between them is clean:

• The machines. Nine nodes — two GPU servers, three virtual machines, two Raspberry Pis and one cloud instance — described in NixOS. The operating system, the packages, the services, the networking.
• The workloads. What runs in the cluster: the deployments, the volumes, the certificates, the cluster’s internal DNS.
• The cloud account. The virtual network, the subnets, the security groups, the storage buckets, the access roles and policies, the serverless functions.
• The edge. The public DNS zones, the tunnels, the authenticated applications, the application firewall and rate-limiting rules.

The most useful boundary, she is the one between the first two and the third: the cloud repository owns everything outside the machine, the NixOS repository owns everything inside it. No bootstrap script, no pre-baked image, no configuration injected at boot. The first one creates the network interface, the static address, the security group and the disk; the second one takes care of the system running on top.

It is a rule that tests itself: if I find myself writing shell inside an infrastructure configuration file, the boundary has just been crossed, and the fix belongs in the other repository.

Adopting an account that already exists, without breaking it

The part the tutorials skip cheerfully: the account, he existed first. He runs. There are things inside him that live services depend on.

The tooling has a mechanism for this — import blocks that say “this resource already exists, here is its identifier, adopt it”. You describe the resource as it truly is, run a plan, and look.

And this is where the rule that saved the project comes in: no plan is applied if it contains a single addition or a single destruction. Zero. A successful adoption plan only makes minor attribute changes — tags being added, essentially. If he proposes to create something, my description does not match what exists and he is about to build a second one. If it proposes to destroy, that is much worse.

This is not decorative caution. In that account, a security group rule that disappear quietly means the cluster goes down. A static IP address replaced means an address lost for good. A storage bucket replaced means data lost. The plan is the last place where those mistakes cost nothing.

The trap that does not say its name

Three traps are worth the detour, because they all have the same shape: the tool was right, and I was the one reading it wrong.

The first. A default route table is not imported by its own identifier. He is imported by the identifier of the virtual network that contains him — the provider goes and looks him up himself. Passing the “right” identifier, the table’s own, made the plan fail with a three-word message, at the very end, naming no resource at all. I accused the state file. The state file had done nothing.

The second. The description field on an access policy is immutable. Not “inadvisable to change”: immutable. By simply omitting it from my code, the plan proposed to replace five policies and all of their attachments — which means, for a moment, a production role without its permissions. The plan, he said so clearly. He just had to be read all the way through.

The third, and this one cost me a real outage. A storage bucket does not change region. Its region is fixed at creation, full stop. To move it, you destroy it and recreate it elsewhere under the same name. Except that after the destruction, the provider, he holds the name hostage — one hour, in my case. No progress bar, no ticket number, nobody to talk to. About fifty minutes of downtime for a one-line change, and a rule written in big letters in the file ever since: create the new one first, destroy the old one last.

Is reality still in agreement?

Code that describes infrastructure is worth nothing if nobody checks that it still corresponds to something. A repository that is right on the day it is written and wrong three weeks later, he is more dangerous than no repository at all, because people trust him.

The problem: the three technologies, they share no common notion of “does reality still match what I declared”. It took one primitive per system.

• For the declared infrastructure: a plan run in “detailed exit code” mode. It exits 0 when there is nothing to do, 2 when there is a difference.
• For the cluster: a comparison between the repository’s manifests and what the server actually holds.
• For the machines: each system carries the git revision number that produced it, and we read it back remotely. If a machine does not report the same revision as the main branch, it has not been rebuilt.

All four run at night, each pushes its result to the same monitoring dashboard, which sends a notification when things go red.

Which brings me to the most useful lesson of the whole exercise: a permanently red check teaches nothing. Mine was red for days, for a stupid reason — the repository pinned a configuration value that the cluster himself rewrites continuously. The check, he was perfectly right to shout. What it had been asked to watch was badly chosen. A permanent red becomes noise, and noise gets muted.

Publishing the code without publishing the lab

Good. The code exists, it is clean, the comments explain every trap above. I wanted to show it. Except that an infrastructure repository, he is literally the floor plan of the house with the location of the locks.

The answer is a sanitizer script per repository, who produce a public snapshot: a transformed copy, published as-is, never synchronized with the original. The principles that came out of it are worth more than the code itself.

The comments are the deliverable; the identifiers are not. This is the central principle and it is counter-intuitive. Nobody learns anything from my account number. But the three-line note above a resource, the one explaining why that field is written out verbatim and what breaks if you remove it — that does not survive being paraphrased. So: every identifier becomes fictional, every comment stays exactly as it is.

Substitute, do not strip. A deleted address, she leaves a hole the reader falls into. An address replaced by a documentation address keeps the shape, the structure, the logic. I even preserve the last octet — the machine that really ends in .7 still ends in .7 in the public version. The reader sees a coherent addressing convention, just not mine.

One single map across the four repositories. The same server carries the same fictional name in all four snapshots, the same fictional address, the same prefix. At first this was only a matter of tidiness. It became structural: it is precisely what makes the four snapshots joinable. More on that in a minute.

Never publish the sanitizer. He excludes himself from his own output. He is the decoder ring: it holds, line by line, the mapping between every fictional name and the real one. Publishing the sanitized version and the lookup table in the same repository would be a complete and perfectly useless piece of work.

Copy only what git tracks. No exclusion list. The tooling, he leaves real state files lying around on disk, and an exclusion list is one typo away from publishing everything. What git does not track does not exist as far as the script is concerned.

The rule that matters: fail loudly

Let us say I make a mistake writing one of my replacement rules. A typo in the domain name to search for, let us say. What happens?

Nothing. The script, he looks for that name, does not find it anywhere, replaces nothing as a result, and finishes normally. No error message — because there was no error: I asked him to look for something that does not exist, and he did not find it. I open the result, it looks clean, I publish. The real domain name is still in there.

That is the dangerous failure mode: it is not a crash, it is a silence. Any cleaning script has this problem, and re-reading it fix nothing, because what you are re-reading is exactly the copy that looks correct.

The solution, she fits in one sentence: the script has to be capable of refusing to finish.

So each script ends with a final step that re-reads its own output and hunts for what should never have been in there — my account number, a resource identifier, a real domain name, an address from my internal network, something shaped like a key, or a filename that still contains a real domain. One single hit and the script stops dead, printing what and on which line.

The nuance that matters: this step fixes nothing. She blocks. That is deliberate — an automatic correction would itself be something that can fail in silence, and we would be right back where we started. And since she runs before any publication, there is never a leak to repair afterwards: there simply was no publication.

For the edge repository, which is essentially the list of every domain name I own, I turned the principle around. Instead of hunting for what is forbidden, the check start from the list of what is permitted: the values the script produces itself, and nothing else. If a DNS record contains anything else, the build blocks. That is stricter than necessary today, and that is exactly the point — the day I add a zone the script does not know about, I want it to blow up right away, not to find out six months later what sailed straight through.

Three bugs deserve naming, because each one would have gone unnoticed without that step.

The dot that was not one. My rules search for exact text, but the tool doing the replacement, he interprets a dot as “any character at all”. So while searching for example.com, I was also replacing example_com — a variable name with nothing to do with any domain. The resulting file no longer compiled. Discovered by accident.

The snake biting its tail. A last-resort rule replaced any forgotten identifier with a dummy one. Except my dummy identifiers had exactly the same shape as the real ones. So the rule found those too, and replaced them with… themselves, all made identical. My cleaning script, he ate his own output.

The domain in the sentence. Best for last. A real domain name survived in a sentence of a documentation file, long after every occurrence in the code had been replaced. My rules covered the code; nobody had thought about the prose. The check, she caught it fifteen minutes before publication. A domain in a sentence leaks exactly as well as a domain in code.

That last bug, he is better watched than described. Here are the few minutes where he got caught — the final check before the repositories went public, right after Ludo said go:

⚠ This is not a live capture: it is a condensed reconstruction, edited afterwards from the real session transcript. The prompts are the ones from the actual session; the timing is compressed and the long waits are cut. And the domain caught on screen is already the fictional name — publishing a demonstration of a leak detector with the real value inside it would have been a little ironic.

Alacritty

What the recording shows and the text does not: the check, she says nothing at all as long as everything is fine. She is invisible for the whole rest of the work. She speaks once, at the worst possible moment — right before it goes public — and what she says is no.

This article, by the way, follow the same rules. The machine names and addresses you have read so far are the ones from the public snapshots, not mine.

What comes next: a view of the lab that draws itself

There is one idea left in the drawer, and it is the one that make all the rest more interesting than a tidying exercise.

An architecture diagram drawn by hand, he is wrong the second somebody changes something. Mine stands up by pure discipline, and discipline, she always end up taking a day off.

So, the project: a view of the lab’s architecture, on this site, generated from the four public snapshots. Not the private repositories — the public ones, and that distinction is the whole design rather than a detail. The sanitizers, they are already the trust boundary, they already have their gate that fails closed, and they already agree on a single map of names and addresses. That is precisely what makes the four snapshots joinable into one picture. A build that reached into the private repositories would put a public site one regex bug away from a real leak; a build that reads only the public ones cannot publish what the gate already refused. Bonus: the site build then needs no secrets at all.

And the part I find most honest: the diagram is only trustworthy when the drift checks are green. That is measurable — it is exactly what the four nightly checks already assert. So the page will show the drift state next to the diagram, instead of claiming a freshness it cannot prove. “Live” is going to mean “regenerated last night, last verification passed at such an hour”. Not “real time”.

That leaves the hard parts, and I am not telling myself stories: four repositories in three different languages, no common schema, and a join key that is the name map imposed by the scripts — which therefore graduates from a convenience to a load-bearing interface. Plus a classic scope trap: the honest first version is a diagram of the four layers and how they stack. Not an automatic reproduction of every resource.

What to take away

Putting existing infrastructure into git, that is not transcription. It is a negotiation with an account who already has opinions, and the only protection that counts is refusing any plan that adds or destroys anything at all.

Publishing it afterwards is a second problem, and the natural reflex — strip out the sensitive things — gives a result full of holes and worth nothing. Substitute rather than strip, keep the comments religiously, and above all: never trust a cleaning script that is not capable of refusing to finish.

The code is really there, if your heart tell you so:

• nixos-iac-public — the nine machines
• k3s-iac-public — the workloads running on them
• aws-iac-public — the cloud account underneath
• cloudflare-iac-public — the edge in front of all of it

The names, addresses, keys and certificates in there are fictional. The comments are the real ones — including the one explaining why you never destroy a storage bucket before creating its replacement.

Bob is going to go check that his four checks are still green. There is one of them that has the red easy.

— Bob