← writing

Sandboxing · microVMs

The LLM wrote the code. Now where do you run it?

I spent a good chunk of last year building a sandbox to run code that a language model wrote. Not code a model suggested to a human who reviewed it — code the agent generated and then executed itself, headless, to get a job done: write some Python, hit a data source, render charts, produce a file. Useful. Also, if you say it plainly: arbitrary attacker-influenced code executing on my infrastructure.

"Attacker-influenced" isn't paranoia. An agent's plan is steered by an LLM, and the LLM's context is full of text nobody on my team wrote — tool output, retrieved documents, user data. Prompt injection is a real, unsolved thing. So I had to assume the code the agent runs could, on a bad day, try to do something hostile, and design for the only question that matters: when it does, what can it touch?

That question has a spectrum of answers. Picking the wrong point on it is how a "sandbox" becomes a word rather than a boundary.

The isolation spectrum

Containers are the default, and for this they're the wrong default. A container shares the host's kernel; namespaces and cgroups are policy bolted onto a kernel the hostile code is still running against. One kernel local-privilege-escalation bug — and the supply is effectively infinite — promotes "stuck in my cgroup" to "root on a multi-tenant host." For code I wrote and trust, containers are great. For code a model wrote under adversarial influence, a container is a standing bet against the entire kernel CVE stream, and I don't want that bet on my balance sheet.

gVisor is a genuine improvement: it runs a user-space kernel that intercepts the guest's syscalls and handles most of them itself, so the real host kernel only sees a small, hardened surface. If your workload has a predictable syscall profile, it's a great pick. My agent runs arbitrary Python, though — an unbounded compatibility and syscall profile, which is exactly the shape gVisor likes least.

Full VMs (QEMU) give the hardware boundary but cost seconds of boot and hundreds of MB of device-emulation surface — most of it a headless code runner never touches.

microVMs are the sweet spot, and Firecracker is the tool. It's a minimal VM monitor that runs on Linux KVM, so the guest gets its own kernel behind a CPU-enforced virtualization boundary — a compromise inside the guest has to beat silicon, not a syscall filter, to reach the host. And it boots in ~125 ms because it emulates only a handful of devices (virtio block, net, vsock, a serial console, an interrupt controller). No BIOS, no PCI, no USB to attack. A hardware boundary at roughly container speed. For running model-authored code, that's not gold-plating — it's the floor.

What a microVM actually costs

Here's what the isolation pitch leaves out. A container hands you an init system, a populated /dev, a package manager, and users. A microVM hands you a bare virtual machine and nothing else. Whatever the workload needs, you build. I paid for the boundary in low-level plumbing, and the bill is itemized.

You supply the kernel and root filesystem as files, and boot them over a tiny API. You PUT a kernel image, a root disk, a machine config, and a network interface at a local socket, then start the instance. That's the easy part.

There's no init and no /dev. I built the guest root filesystem from scratch — a tiny musl userland — and wrote PID 1 by hand: mount /proc, /sys, /dev, bring up the network, then drop to the workload. A representative gotcha:

#!/bin/sh
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount -t devtmpfs dev /dev
# mounting devtmpfs just shadowed the /dev/fd symlinks the
# rootfs shipped, so recreate them AFTER the mount, or
# <(...) in bash silently breaks:
ln -sf /proc/self/fd /dev/fd

I lost real time to exactly that: bake the /dev/fd symlinks into the image, mount devtmpfs, watch it shadow them with an empty directory, watch process substitution break with a useless error. If you use busybox, you also have to create the applet symlinks yourself or half your commands don't exist on PATH.

The workload has to run non-root. The agent runtime I used refuses its headless, don't-stop-to-ask mode as root — and an agent that stops to ask can't run unattended. So the code runs as an unprivileged user. But a from-scratch rootfs has no baselayout for adduser to work with, so I wrote /etc/passwd and /etc/group directly and launched with su -p to preserve the environment.

No shared filesystem — at all. Firecracker deliberately omits virtio-fs and 9p, so you can't mount a host directory or object storage into the guest. Data comes in as a block device. So every run I mkfs an ext4 disk, bake the workspace into it, boot with it attached, and read the results back out afterward. And because the host had no spare loop devices, it all had to be loopless:

# populate a filesystem image from a directory at
# creation — no mount:
mkfs.ext4 -d ./workspace-tree -F work.ext4 12G
# ... boot the microVM with work.ext4 as a data disk ...
# read outputs back without ever mounting the image:
debugfs -R "rdump /out ./results" work.ext4

mkfs.ext4 -d to write and debugfs rdump to read is the loopless idiom that unblocked the whole thing.

Cold start and disk are yours to manage now. Copying a ~1 GB root filesystem per run was too slow, so I boot a shared base read-only and attach a small scratch disk as a writable overlay via pivot_root. One tuition-worthy trap: the busybox switch_root helper demands the old root be a ramfs and aborts on an ext4 base — the guest exits and the VM just reboots in a few seconds, so the workload silently never runs and you get no error, only a suspiciously fast "success." pivot_root has no such requirement. And an early config bug that dragged stale data into every run filled the disk and surfaced as a clean exit that quietly produced nothing. When you own the whole machine, you own ENOSPC and the confusing non-failures too.

You also need the hardware. Firecracker needs KVM (/dev/kvm), and a lot of cloud instances are themselves guests that don't expose the CPU's virtualization extensions — so no /dev/kvm, no microVM. You either pay for bare metal or pick a newer nested-virtualization-capable instance. Nested virt has a small performance tax; for a multi-minute job it's noise.

The bonus the VM boundary buys: secrets never enter the box

A hardware boundary contains a breach, but the best breach is a boring one. So I kept secrets out of the sandbox entirely. The code needs results from a data source, not the credentials to it — so at job start I mint a short-lived, signed, per-job token and put only that inside the guest. When the code needs data, it calls a proxy on the trusted side:

# inside the sandbox — no data-source creds here, ever
r = httpx.post(
    "http://proxy.internal/invoke",
    headers={"Authorization": f"Bearer {os.environ['JOB_TOKEN']}"},
    json={"source": "warehouse", "op": "query", "q": q},
)

The proxy derives who's asking from the signed token — not a header the guest could forge — runs the real query on the trusted side, and returns only rows. A full compromise of the sandbox nets an attacker some already-authorized results and a token that expires in minutes. Nothing durable.

What to steal from this

• —Match the boundary to the threat. Code you wrote → a container is fine. Code a model wrote under adversarial influence → you want a boundary that survives a host-kernel bug. That's hardware virtualization.
• —microVMs make that cheap. VM-grade isolation and fast/dense/cheap stopped being opposites the moment Firecracker existed.
• —Price in the userland. No shared FS means you mkfs a disk per run. No init means you write PID 1. No users means you write /etc/passwd. The isolation is the free part.
• —Keep long-lived secrets out of the sandbox. Hand it a scoped, expiring token and proxy the privileged work. A hard boundary plus a boring breach is the whole game.