Run a Nix remote builder and binary cache on exe.dev

How it works#

• Nix on my-nix-builder, installed the official way, with exedev as a trusted user. Clients connect with ssh-ng://my-nix-builder.exe.xyz, the Nix daemon protocol over SSH. With builders-use-substitutes, the builder downloads dependencies from cache.nixos.org itself, instead of your laptop uploading them.
• Harmonia serves /nix/store on port 8000, so https://my-nix-builder.exe.xyz/ is a binary cache of everything the builder has built or downloaded. It signs each path with the VM's key; clients trust that key and nothing else from it.
• Garbage collection keeps the disk from filling: Nix collects when a build runs low on space, and harmonia-gc runs nightly, keeping anything from the last 14 days.

A machine can use the builder, the cache, or both. A laptop usually wants both; a CI job that only needs what was already built wants the cache alone, with a read-only token.

Set up the builder#

ssh exe.dev new --name=my-nix-builder --cpu=8 --memory=16GB --disk=200GB
ssh my-nix-builder.exe.xyz

nproc; free -h; df -h /

curl --proto '=https' --tlsv1.2 -sSfL https://nixos.org/nix/install | sh -s -- --daemon --yes

for p in nix nix-daemon nix-store; do
  sudo ln -sf /nix/var/nix/profiles/default/bin/$p /usr/local/bin/$p
done
sudo tee -a /etc/nix/nix.conf >/dev/null <<'EOF'
experimental-features = nix-command flakes
trusted-users = root exedev
max-jobs = auto
cores = 0
secret-key-files = /etc/nix/cache-key.sec
min-free = 20G
max-free = 60G
EOF

sudo -i sh -c "umask 077; nix key generate-secret --key-name $(hostname)-1 > /etc/nix/cache-key.sec"
sudo -i sh -c 'nix key convert-secret-to-public < /etc/nix/cache-key.sec > /etc/nix/cache-key.pub'
sudo systemctl restart nix-daemon
cat /etc/nix/cache-key.pub; echo

nix store info --store daemon
sudo -i nix config show system-features

sudo mkdir -p /etc/systemd/system/nix-daemon.service.d
printf '[Service]\nExecStartPre=-/bin/chmod 0666 /dev/kvm\n' \
  | sudo tee /etc/systemd/system/nix-daemon.service.d/kvm.conf
sudo systemctl daemon-reload && sudo systemctl restart nix-daemon

sudo -i nix profile add --profile /nix/var/nix/profiles/harmonia nixpkgs#harmonia
sudo ln -sf /nix/var/nix/profiles/harmonia/bin/harmonia-gc /usr/local/bin/harmonia-gc
sudo mkdir -p /etc/harmonia
sudo tee /etc/harmonia/harmonia.toml >/dev/null <<'EOF'
bind = "[::]:8000"
priority = 50
EOF
sudo tee /etc/systemd/system/harmonia.service >/dev/null <<'EOF'
[Unit]
Description=Harmonia Nix binary cache
After=network-online.target nix-daemon.socket
Wants=network-online.target

[Service]
Type=notify
ExecStart=/nix/var/nix/profiles/harmonia/bin/harmonia-cache
Environment=CONFIG_FILE=/etc/harmonia/harmonia.toml
Environment=SIGN_KEY_PATHS=%d/sign-key
LoadCredential=sign-key:/etc/nix/cache-key.sec
DynamicUser=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
NoNewPrivileges=yes
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now harmonia
curl -s localhost:8000/nix-cache-info

sudo tee /etc/systemd/system/harmonia-gc.service >/dev/null <<'EOF'
[Unit]
Description=Nix garbage collection (harmonia-gc)

[Service]
Type=oneshot
Environment=PATH=/nix/var/nix/profiles/default/bin:/usr/bin:/bin
ExecStart=/nix/var/nix/profiles/harmonia/bin/harmonia-gc --ensure-free 25%% --keep-recent 14d
EOF
sudo tee /etc/systemd/system/harmonia-gc.timer >/dev/null <<'EOF'
[Unit]
Description=Nightly Nix garbage collection

[Timer]
OnCalendar=03:15
RandomizedDelaySec=30m
Persistent=true

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now harmonia-gc.timer
sudo systemctl start harmonia-gc.service
journalctl -u harmonia-gc -n 3 --no-pager -o cat

out=$(nix build --no-link --print-out-paths --impure --expr \
  'with import (builtins.getFlake "nixpkgs") {}; runCommand "cache-check" {} "date > $out"')
curl -s localhost:8000/$(basename $out | cut -c1-32).narinfo | grep -E '^(StorePath|Sig):'

After Deploy, Shelley has done all of the above and shows you the builder's public key. The rest happens on the machines that will use it.

Connect a Linux machine#

This is for any Linux machine with Nix's multi-user install (a workstation, a server, another exe.dev VM). The Nix daemon, which runs as root, does the SSH and the downloads, so the key and the token are root's.

On the machine, make an SSH key for the daemon and show it:

sudo install -d -m 700 /root/.ssh
sudo ssh-keygen -q -t ed25519 -N '' -C "nix-daemon@$(hostname)" -f /root/.ssh/nix-builder
sudo cat /root/.ssh/nix-builder.pub

On your computer, let that key into the builder and nowhere else, and make a token for the cache (paste the public key between the quotes):

ssh exe.dev ssh-key add --vm=my-nix-builder 'ssh-ed25519 AAAA... nix-daemon@workstation'
ssh exe.dev ssh-key generate-api-key --vm=my-nix-builder --label=nix-cache-workstation --exp=1y

The second command prints a token starting with exe1.. It gets through the HTTPS proxy to this VM only, and Harmonia only serves reads, so it can't change anything.

Back on the machine, connect once as root, which records the host key and proves the key works, then tell Nix about the builder, the cache, and the token. Put in your builder's public key, the token, and the system-features from setup (drop kvm if the builder lacks it):

sudo ssh -i /root/.ssh/nix-builder -o StrictHostKeyChecking=accept-new my-nix-builder.exe.xyz nix --version
echo 'ssh-ng://my-nix-builder.exe.xyz x86_64-linux /root/.ssh/nix-builder 8 1 nixos-test,benchmark,big-parallel,kvm - -' \
  | sudo tee /etc/nix/machines
printf 'Cache token: '; read -rs T; echo
printf 'machine my-nix-builder.exe.xyz\nlogin nix\npassword %s\n' "$T" | sudo install -m 600 /dev/stdin /etc/nix/netrc
unset T
sudo tee -a /etc/nix/nix.conf >/dev/null <<'EOF'
extra-experimental-features = nix-command flakes
builders-use-substitutes = true
extra-substituters = https://my-nix-builder.exe.xyz
extra-trusted-public-keys = my-nix-builder-1:KISg07I/HaZWK0r8bXNquX6lfQ79MAVlLnk6NVoeHgY=
netrc-file = /etc/nix/netrc
EOF
sudo systemctl restart nix-daemon

The nix --version prints the builder's Nix version. In the machines line, 8 is how many builds to send at once (the builder's vCPUs) and 1 its speed factor; /etc/nix/machines is where Nix looks by default. Check both halves:

sudo -i nix store info --store 'ssh-ng://my-nix-builder.exe.xyz?ssh-key=/root/.ssh/nix-builder'
curl -s --netrc-file <(sudo cat /etc/nix/netrc) https://my-nix-builder.exe.xyz/nix-cache-info

The first says Trusted: 1; the second prints the cache info from setup. Now build something that isn't in any cache, on the builder only:

nix build --max-jobs 0 -L --impure --expr \
  'with import (builtins.getFlake "nixpkgs") {}; hello.overrideAttrs { NONCE = "'$RANDOM'"; }'

The log says building '/nix/store/…-hello-2.12.3.drv' on 'ssh-ng://my-nix-builder.exe.xyz'. --max-jobs 0 means "never build here"; without it, Nix builds locally while it has free slots and sends the rest. Put max-jobs = 0 in nix.conf to make that the default on a laptop.

NixOS and nix-darwin#

On NixOS, the same settings go in your configuration (nix-darwin takes the same options); the key and netrc files stay where you made them:

{
  nix.distributedBuilds = true;
  nix.buildMachines = [{
    hostName = "my-nix-builder.exe.xyz";
    protocol = "ssh-ng";
    systems = [ "x86_64-linux" ];
    sshKey = "/root/.ssh/nix-builder";   # /var/root/.ssh/nix-builder on a Mac
    maxJobs = 8;
    supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
  }];
  nix.settings = {
    builders-use-substitutes = true;
    extra-substituters = [ "https://my-nix-builder.exe.xyz" ];
    extra-trusted-public-keys = [ "my-nix-builder-1:KISg07I/HaZWK0r8bXNquX6lfQ79MAVlLnk6NVoeHgY=" ];
    netrc-file = "/etc/nix/netrc";
  };
}

Connect a Mac#

A Mac builds its own aarch64-darwin (or x86_64-darwin) packages and sends x86_64-linux ones to the builder: NixOS systems, Docker images from dockerTools, anything your servers run. The daemon runs as root, whose home on macOS is /var/root.

On the Mac, make the daemon's key and show it:

sudo install -d -m 700 /var/root/.ssh
sudo ssh-keygen -q -t ed25519 -N '' -C "nix-daemon@$(hostname -s)" -f /var/root/.ssh/nix-builder
sudo cat /var/root/.ssh/nix-builder.pub

Add it and make a token exactly as in Connect a Linux machine (ssh exe.dev ssh-key add --vm=my-nix-builder … and generate-api-key), then:

sudo ssh -i /var/root/.ssh/nix-builder -o StrictHostKeyChecking=accept-new my-nix-builder.exe.xyz nix --version
echo 'ssh-ng://my-nix-builder.exe.xyz x86_64-linux /var/root/.ssh/nix-builder 8 1 nixos-test,benchmark,big-parallel,kvm - -' \
  | sudo tee /etc/nix/machines
printf 'Cache token: '; read -rs T; echo
printf 'machine my-nix-builder.exe.xyz\nlogin nix\npassword %s\n' "$T" | sudo tee /etc/nix/netrc >/dev/null
sudo chmod 600 /etc/nix/netrc; unset T

Then add the same five lines as on Linux (from extra-experimental-features to netrc-file) to Nix's configuration and restart the daemon:

• Nix from nixos.org: append them to /etc/nix/nix.conf, then sudo launchctl kickstart -k system/org.nixos.nix-daemon.
• Determinate Nix: append them to /etc/nix/nix.custom.conf (its nix.conf is managed), then sudo launchctl kickstart -k system/systems.determinate.nix-daemon.
• nix-darwin: use the module settings and darwin-rebuild switch.

Try it with a Linux build:

nix build -L --impure --expr \
  'with import (builtins.getFlake "nixpkgs") { system = "x86_64-linux"; }; runCommand "where" {} "uname -sm > $out"'
cat result

Linux x86_64: it was built on the VM and copied back.

Use the cache without the builder#

A machine that only needs what the builder already has (a deploy target, a teammate's laptop, a CI job for pull requests) needs no SSH key: give it a token and the same nix.conf lines without the machines file. Anything it builds itself stays on that machine.

For exe.dev VMs, skip the token: a VM-to-VM integration reaches the cache with a key that exe.dev holds and injects, so it's on no VM. On your computer:

ssh exe.dev integrations add http-proxy --name nix-cache \
  --target https://my-nix-builder.exe.xyz/ --peer --attach tag:nix
ssh exe.dev tag my-app nix

On every VM tagged nix, the cache is https://nix-cache.int.exe.xyz:

extra-substituters = https://nix-cache.int.exe.xyz
extra-trusted-public-keys = my-nix-builder-1:KISg07I/HaZWK0r8bXNquX6lfQ79MAVlLnk6NVoeHgY=

Making the cache public (ssh exe.dev share set-public my-nix-builder) opens the cache to everyone, and never the builder.

Use it from CI#

CI is where a builder and cache earn their keep: every job starts with an empty store, and without help it rebuilds whatever cache.nixos.org doesn't have. The same key, token and settings work from any CI that runs Nix on Linux.

GitHub-hosted runners#

Make a key and a token for CI on your computer, and store both as repository secrets (this uses the GitHub CLI; Settings → Secrets and variables → Actions works too):

ssh-keygen -q -t ed25519 -N '' -C nix-ci -f nix-ci
ssh exe.dev ssh-key add --vm=my-nix-builder < nix-ci.pub
gh secret set NIX_BUILDER_SSH_KEY < nix-ci
ssh exe.dev ssh-key generate-api-key --vm=my-nix-builder --label=nix-cache-ci --exp=1y --json \
  | jq -r .token | gh secret set NIX_CACHE_TOKEN
rm nix-ci nix-ci.pub

Then, in a workflow:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: cachix/install-nix-action@v31
        with:
          extra_nix_config: |
            builders = ssh-ng://my-nix-builder.exe.xyz x86_64-linux /root/.ssh/nix-builder 8 1 nixos-test,benchmark,big-parallel - -
            builders-use-substitutes = true
            extra-substituters = https://my-nix-builder.exe.xyz
            extra-trusted-public-keys = my-nix-builder-1:KISg07I/HaZWK0r8bXNquX6lfQ79MAVlLnk6NVoeHgY=
            netrc-file = /etc/nix/netrc
      - name: Connect to the Nix builder
        env:
          SSH_KEY: ${{ secrets.NIX_BUILDER_SSH_KEY }}
          CACHE_TOKEN: ${{ secrets.NIX_CACHE_TOKEN }}
        run: |
          sudo install -d -m 700 /root/.ssh
          printf '%s\n' "$SSH_KEY" | sudo install -m 600 /dev/stdin /root/.ssh/nix-builder
          sudo ssh -i /root/.ssh/nix-builder -o StrictHostKeyChecking=accept-new my-nix-builder.exe.xyz true
          printf 'machine my-nix-builder.exe.xyz\nlogin nix\npassword %s\n' "$CACHE_TOKEN" \
            | sudo install -m 600 /dev/stdin /etc/nix/netrc
      - run: nix build --max-jobs 0 -L .#default

Everything the job builds is built on my-nix-builder and stays in its cache, so the next job, on any branch, downloads it.

The builder key is root on the builder, so don't hand it to code you don't trust. For pull requests from forks, which can't read secrets anyway, leave the builder out; for pull requests from people with write access you trust, it's the same as giving them ssh to the VM. A job that should only read the cache uses NIX_CACHE_TOKEN alone.

Jobs on exe.dev sandboxes#

With GitHub Actions on exe.dev, each job runs on a fresh copy of a base VM. Install Nix on the base, with the cache lines from Use the cache without the builder pointing at the nix-cache integration, and attach that integration by the ci tag the job VMs carry (--attach tag:ci). Then:

• Every job VM starts with the base's /nix/store, so it begins warm, and fetches the rest from the cache with no token on any VM.
• For builds the base doesn't have, add the builder as on GitHub's runners, with the key from a GitHub secret. Don't put the key on the base: every job VM would have it.

Refresh the base's store now and then (ssh my-ci-base.exe.xyz 'nix build .#default && sync' from a checkout), as with any other warm cache.

Build times#

Measured on 2026-11-04 with Nix 2.35.4 and nixpkgs 0bb2eed, between a 2 vCPU / 4 GB exe.dev VM as the client (about a small laptop's worth) and an 8 vCPU / 32 GB builder. Each package was rebuilt from source with every dependency already in cache.nixos.org, by adding an unused environment variable to it. Times are nix build wall-clock, after evaluation.

The builder column includes copying the result back to the client. Small packages gain little: hello spends its time in a configure script that runs on one core either way. The bigger the build, the more of it runs in parallel and the more the builder's cores pay off, but no package uses them all the whole time (Python's test suite, cmake's bootstrap), so 4× the cores gave 2 to 3× the speed. Several builds at once, as when a NixOS system or a CI matrix needs many packages, use them better.

"From the cache" is the second machine to want the same output: one download from the builder instead of any build, timed here between two exe.dev VMs. From your own network, it goes at your connection's speed.

The first remote build of a package also downloads its build dependencies to the builder (the Rust toolchain for ripgrep, say), once: 81 s instead of 57 s. A NixOS VM test (runNixOSTest) that boots one machine took 35 s on the builder with KVM.

Keep the disk from filling#

The store only grows: every build adds its outputs and the dependencies it downloaded. Two things trim it.

• During builds, Nix itself: when a build finds less than min-free (20 GB) free, it deletes garbage until max-free (60 GB) is free.
• Every night, harmonia-gc --ensure-free 25% --keep-recent 14d: if less than a quarter of the disk is free, it deletes garbage until a quarter is, but never a path added in the last 14 days.

"Garbage" is anything no GC root refers to, and on a builder that is almost everything: the outputs it built for your clients went back to them, and nothing on the builder keeps them. That's fine for a cache, as long as the cache is mostly recent work, which --keep-recent ensures. Two things to know:

• Collection goes by when a path arrived in the store, not when it was last downloaded. A package built a month ago and fetched every day can still go; the next client that needs it builds it again, and it's back.
• To keep something no matter what (a release, the current production system), add a GC root for it on the builder: sudo -i nix build --out-link /nix/var/nix/gcroots/releases/app-1.4 'github:you/app/v1.4'. Delete the link to let it go.

See what's there and what would go, on the VM:

df -h /
sudo harmonia-gc --dry-run | tail -1

If the disk is often full, grow it from your computer with ssh exe.dev resize my-nix-builder --disk=400GB, then sudo resize2fs $(findmnt -no SOURCE /) on the VM, or keep fewer days. exe.dev bills disk by what's used, not its size.

What it costs#

The builder is an ordinary VM on your plan. On Personal, all your VMs share one capacity, so an 8 vCPU builder on the $75-a-month plan uses all of it while it builds and none of it while idle, and the plan's 400 GB of disk covers the store. Cache downloads leave exe.dev and count toward the plan's 200 GB of outbound bandwidth a month; a team pulling big closures every day should keep an eye on ssh exe.dev billing usage.

For builds that only happen sometimes, a sandbox is billed by the second: a 16 vCPU / 32 GiB one is $0.84 an hour. Its store is gone when you delete it, though, so keep the cache on a regular VM.

Back up and restore#

The one thing to keep is the signing key. Every client trusts my-nix-builder-1, and a new key means editing every client. On your computer:

ssh my-nix-builder.exe.xyz 'sudo cat /etc/nix/cache-key.sec' > my-nix-builder-1.sec
chmod 600 my-nix-builder-1.sec

Store it as you'd store any secret. The store itself is a cache: what you lose with it, the builder rebuilds or downloads again. To move the builder, copy the whole VM (ssh exe.dev cp my-nix-builder my-nix-builder-2), or set up a new one and restore the key to /etc/nix/cache-key.sec (mode 600, owned by root) before starting Harmonia.

If the key leaks, make my-nix-builder-2 as in setup, replace the public key on every client, and restart Harmonia. Paths signed only by the old key stop being trusted, which is the point.

Update#

On the VM:

• Nix: sudo -i nix upgrade-nix, then sudo systemctl restart nix-daemon. Clients and builder don't need the same version, but keep them within a few releases.
• Harmonia: sudo -i nix profile upgrade --profile /nix/var/nix/profiles/harmonia --all, then sudo systemctl restart harmonia.
• Tokens and keys: the cache tokens above expire in a year. Make a new one with generate-api-key and replace /etc/nix/netrc (or the CI secret). ssh exe.dev ssh-key list shows every key and token; ssh exe.dev ssh-key remove <name> revokes one, such as a lost laptop's nix-daemon@… key.

Troubleshooting#

On the client, add -L -v to nix build to see which machine it picks and why. On the builder, journalctl -u nix-daemon -f and journalctl -u harmonia -f.

• Failed to find a machine for remote build! The derivation needs a system or feature the machines line doesn't list. A Linux build from a Mac needs x86_64-linux; a NixOS test needs kvm and nixos-test, which only a builder with /dev/kvm can offer.
• cannot build on 'ssh-ng://…': … Permission denied (publickey). The daemon's key isn't added, or you added your own instead. Test as root: sudo ssh -i /root/.ssh/nix-builder my-nix-builder.exe.xyz true. ssh exe.dev ssh-key list shows the keys scoped to the VM.
• Host key verification failed. Root never connected before; run the sudo ssh … -o StrictHostKeyChecking=accept-new line once.
• nix-daemon: command not found. The symlinks in /usr/local/bin on the builder are missing; recreate them.
• … is not trusted or cannot add path … lacks a signature. exedev isn't in trusted-users on the builder, or the daemon wasn't restarted after the change.
• The cache is never used (nix build builds everything again). Check the token: curl -s --netrc-file <(sudo cat /etc/nix/netrc) https://my-nix-builder.exe.xyz/nix-cache-info must print the cache info, not an HTML login page (missing or expired token). Then check that extra-trusted-public-keys matches /etc/nix/cache-key.pub exactly; a wrong key shows as ignoring substitute … because it's not signed by any of the keys in 'trusted-public-keys'.
• A path you just built isn't found in the cache (there is no substituter that can build it), though curl finds its .narinfo. The client asked for it before it existed and remembers the miss for an hour. Add --option narinfo-cache-negative-ttl 0 to that one nix build.
• A NixOS test is very slow, and its log (nix log on the builder) says failed to initialize kvm: Permission denied and falling back to tcg. /dev/kvm isn't open to builds; see the kvm.conf step in Install Nix.
• No space left on device during a build. Run sudo harmonia-gc --ensure-free 25% on the builder, then see Keep the disk from filling.