iacthing Get in touch

Every host · one repository · one command

A fleet, without the cluster.

Four machines on Kubernetes means a cluster to keep alive, a stack of add-ons to make it usable, and your app rewritten as YAML. iacthing is one repository of NixOS hosts, and a command that makes every machine match it.

See it side by side What it holds you to

Act I

The Kubernetes way.

You have one app, a database and four machines. The industry standard answer starts with a cluster, and the app comes last.

01 · A cluster first

Before the app, a cluster.

A control plane, a network plugin, and nodes that say NotReady until you've found out which one you forgot.

~/cluster
$ kubeadm init --pod-network-cidr=10.244.0.0/16
Your Kubernetes control-plane has initialized successfully!
$ kubectl apply -f calico.yaml
customresourcedefinition.apiextensions.k8s.io/… created ×24
$ kubectl get nodes
node-1 Ready control-plane
node-2 NotReady <none>

02 · Make it usable

Then the add-ons.

Ingress, certificates, DNS, secrets, monitoring and a GitOps controller. Each is its own chart, its own values file, and its own upgrade path.

~/cluster
$ helm install ingress-nginx ingress-nginx/ingress-nginx -f ingress.yaml
$ helm install cert-manager jetstack/cert-manager -f certs.yaml
$ helm install external-dns bitnami/external-dns -f dns.yaml
$ helm install sealed-secrets sealed-secrets/sealed-secrets
$ helm install monitoring prometheus-community/kube-prometheus-stack
$ helm install argocd argo/argo-cd -f argocd.yaml
# your app is still not installed

03 · The app, as YAML

Your app, rewritten.

A Deployment, a Service and an Ingress with the annotations the ingress controller wants, templated so the chart can fill in what it knows.

deployment.yaml lines 1–16
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
spec:
replicas: 2
template:
spec:
containers:
- name: app
image: ghcr.io/you/app:{{ .Values.image.tag }}
envFrom:
- secretRef:
name: app-env
ports:
- containerPort: 4000

04 · Then it breaks

And it won't start.

The pod restarts forever. The reason is three commands deep: the sealed secret was sealed for the other cluster.

~/cluster
$ kubectl get pods
app-7d9f8c6b5-x2k4p 0/1 CrashLoopBackOff 14
$ kubectl describe pod app-7d9f8c6b5-x2k4p
Warning Failed secret "app-env" not found
$ kubectl logs -n sealed-secrets deploy/sealed-secrets
no key could decrypt secret (app-env)

05 · To hold four machines

All of it, to run one app.

Once it works, this is what's running before your app gets a single pod.

~/cluster
$ kubectl get pods -A --no-headers | wc -l
217
$ kubectl get namespaces --no-headers | wc -l
14
# on four machines. one of the pods is yours.

That was one app.

commands
14
add-ons before the app
7
YAML files to keep
7
pods, for four machines
217

Here's the same fleet, with iacthing.

Act II

The iacthing way.

One repository declares every host, every service and every setting. You change the repository, and the fleet becomes it.

01 · A host is a file

Everything a machine is.

Written once. The next machine is a copy and an edit, not a new project, and it's still plain NixOS.

hosts/web-01/default.nix lines 1–12
{ ... }:
{
imports = [
../../services/caddy
../../services/app
];
networking.hostName = "web-01";
networking.firewall.allowedTCPPorts = [ 80 443 ];
services.openssh.enable = true;
sops.secrets.app-env.owner = "app";
}

02 · A service is a module

Services are files too.

A database is a module a host imports, and its backup is declared beside it. Nothing about it lives only on a machine.

services/postgres/default.nix lines 1–17
{ config, ... }:
{
services.postgresql = {
enable = true;
ensureDatabases = [ "app" ];
ensureUsers = [
{ name = "app"; ensureDBOwnership = true; }
];
};
services.restic.backups.postgres = {
paths = [ "/var/backup/postgresql" ];
repository = "s3:s3.example/backups";
passwordFile = config.sops.secrets.restic.path;
timerConfig.OnCalendar = "daily";
};
}

03 · One command

The fleet becomes the repository.

Every machine is built from the same flake and switched over ssh. A host that already matches is left alone.

~/fleet
$ iacthing switch --all
web-01 3 services ······ 41 s
web-02 0 services ······ already declared
build-01 3 services ······ 36 s
db-01 3 services ······ 44 s
✓ 4 machines · 2 m 39 s

04 · When it breaks

It fails before it ships.

A missing secret is a build error on your laptop, not a pod restarting on a node. And a running box is ordinary systemd and a journal.

~/fleet
$ iacthing switch web-01
✗ sops: secret app-env is not in secrets/web-01.yaml
$ sops secrets/web-01.yaml
$ iacthing switch web-01
✓ web-01 · app running under systemd

Act III

What it holds you to.

The rules are the product. They're what make the tenth machine as boring as the first.

  1. 1

    Declarative

    If a setting matters, it's written down. A change made by hand on a running box is a bug, not a fix.

  2. 2

    Secrets, once

    Every project goes through sops, so a key lives in one place and nothing is copied between machines by hand.

  3. 3

    Rebuildable

    Hardware is disposable. Everything needed to make a box again is in the repository, and switching twice ends in the same state.

Four machines, both ways
Measure Kubernetes iacthing
Before the app runs a cluster and its add-ons nothing
What describes the fleet charts, values and manifests one repository
Upgrades one path per add-on one flake.lock
A missing secret CrashLoopBackOff a build error
Debugging pods, events, controller logs systemd and a journal

Running a few machines?

Tell me what's on them. iacthing is what our own servers are declared in, hostingthing among them.

Get in touch