# Proxmox runners

Connect a Proxmox VE cluster and run Linux jobs in disposable QEMU virtual machines.

The Proxmox connector lets Mirage run Linux jobs in disposable QEMU virtual machines on
your Proxmox VE cluster. It clones a prepared template for each accepted job and destroys
the clone after completion or cancellation. Proxmox capacity appears under Mirage’s
**Local infrastructure** provider; workflows and fallback do not need a Proxmox-specific
runner label.

Setup is two commands on two machines: one inside the VM that becomes your template, and
one on the small Linux host that connects Mirage to your Proxmox API. In
[Machines](https://mirageci.com/machines), **Pair new machine**, then **Linux** and **Proxmox**, shows both.

> **Your template brings the runner software.**
>
> Mirage installs its own handoff into the template, but the template must already contain
> the Mirage runner software, Docker Engine, and the QEMU guest agent. Setup checks for them
> and stops if they are missing; it never installs the runner itself.

## When to use Proxmox

Choose Proxmox when you operate a Proxmox VE cluster and want a QEMU VM boundary for each
Linux job. The connector is a lightweight service that talks to Mirage and the Proxmox
HTTPS API; it does not need Docker, and its own CPU and memory do not limit how many jobs
it runs. Run it in one of two places:

- **On the Proxmox node.** On a single node, install it directly on the host. No always-on
  VM is needed.
- **On a small connector VM or machine.** When it is a Proxmox VM, enable **Start at boot**
  for it so the connector returns after a node reboot. Only the connector’s own VM gets
  Start at boot; the template and every job clone keep it off.

Use one connector for each template architecture and target node. Workloads request their own sizes; older connectors use
the same source template, target node, CPU count, memory, and disk contract. The connector
supports one template and one target node; it does not provide HA placement, multi-node
capacity discovery, automatic template creation, or a warm VM pool.

## Prepare a VM template

Build a Linux x64 or arm64 VM with systemd that has:

- the same runner version and software payload as the Mirage runner image, including
  Docker Engine and `/usr/local/bin/mirage-entrypoint`;
- `qemu-guest-agent` installed;
- outbound HTTPS and GitHub Actions connectivity;
- no persistent job workspace, repository credential, or copied production SSH host key;
- a root disk at least as large as the connector’s runner disk setting.

Inside the VM, as a user with `sudo`, run:

Template VM · install the Mirage handoff:

```bash
bash <(curl -fsSL https://mirageci.com/bootstrap/proxmox-template)
```

Setup checks the VM, verifies a signed Mirage release, and installs the one-time
configuration handoff: a small service that reads a job’s runner configuration from tmpfs,
deletes it before starting the runner, and reports only readiness and the final exit code
to the connector. It disables the guest’s own `docker.service` and `docker.socket`, because
the runner starts an isolated Docker daemon, and enables the guest agent and the handoff.
It also empties `/etc/machine-id`, so each clone generates its own. Running it again is
safe.

Then finish the template in Proxmox:

1. Shut the VM down, and don’t start it again before converting it. A boot gives the VM a
   machine ID that every clone then shares, and clones that run at once are handed the same
   DHCP address. If it does start, run setup again before shutting it down.
2. In its **Options**, enable **QEMU Guest Agent** and disable **Start at boot** and
   **Protection**, so interrupted clones stay safe to destroy.
3. Add the tag `mirage-template`, then convert the VM to a template.
4. Note the template’s node and VMID for the connector.

## Grant Proxmox access

Create a dedicated Proxmox API token and, preferably, a dedicated `mirage` pool. Scope the
token to audit the source template and VM inventory; clone and configure VMs in the target
pool and storage; use the selected network; start, stop, and delete those VMs; and call the
QEMU Guest Agent. Exact ACL paths depend on your pool, storage, and SDN layout. Do not use
an administrator token.

On Proxmox VE 9, this grant works for one node with a `mirage` pool, job disks on
`local-zfs`, and a plain Linux bridge (the built-in `localnetwork` zone). Substitute your
node, storage, zone, and template VMID:

Proxmox node · create the pool, token, and grant:

```bash
pveum pool add mirage
pvesh set /pools/mirage --vms <template-vmid>
pveum user add mirage@pve --comment "Mirage Proxmox connector"
pveum user token add mirage@pve connector --privsep 1  # prints the secret once

pveum role add MirageRunnerConnector --privs "Pool.Audit,VM.Allocate,VM.Audit,\
VM.Clone,VM.Config.CPU,VM.Config.Memory,VM.Config.Options,VM.GuestAgent.Audit,\
VM.GuestAgent.FileRead,VM.GuestAgent.FileWrite,VM.PowerMgmt"
pveum role add MirageTaskAudit --privs Sys.Audit
pveum role add MirageSDNUse --privs SDN.Use

# A privilege-separated token gets the intersection of user and token ACLs.
for who in "--users mirage@pve" "--tokens mirage@pve!connector"; do
  pveum acl modify /pool/mirage --roles MirageRunnerConnector $who
  pveum acl modify /nodes/<node> --roles MirageTaskAudit --propagate 0 $who
  pveum acl modify /sdn/zones/localnetwork --roles MirageSDNUse $who
  pveum acl modify /storage/local-zfs --roles PVEDatastoreUser $who
done
```

Proxmox VE 8 has no `VM.GuestAgent.*` privileges; grant `VM.Monitor` there instead. Keep
the token ID (`mirage@pve!connector`) and its secret for the next step. TLS certificate
verification cannot be disabled, so if Proxmox uses a private certificate authority, keep
its PEM certificate too.

## Set up the connector

On the Proxmox node, or on a dedicated Linux x64 or arm64 host with systemd and HTTPS
access to `mirageci.com` and your Proxmox API, run as a user with `sudo`, or as root on a
node without it:

Connector host · install, check, and pair:

```bash
bash <(curl -fsSL https://mirageci.com/bootstrap/linux) --proxmox
```

Setup installs the signed Mirage agent, then asks for:

| Setting | Default |
| --- | --- |
| Proxmox API URL (`https://…:8006`) | required |
| API token ID and secret | required |
| Template node and VMID | required |
| Target node | the template node |
| Concurrent jobs | `1` |
| First VMID of Mirage’s reserved range | `900000` |
| Pool | none; a dedicated pool is recommended |
| Private CA certificate | none |

On the node itself, use `https://127.0.0.1:8006` as the API URL and
`/etc/pve/pve-root-ca.pem` as the CA certificate; the node certificate covers the local
address. The token secret is read without echo and stored only in
`/etc/mirage/proxmox-token`, readable by the Mirage service account alone. The reserved
VMID range starts at the first VMID and holds one ID per concurrent job; keep the template
outside it.

Before pairing, setup runs the connector’s checks: API access, the template flag and tag,
the QEMU Guest Agent option, and the reserved VMID range. If a check fails, fix the value in
`/etc/mirage/agent.env` (with `sudoedit`, or directly as root) and run setup again; the host does not appear in
Mirage until the checks pass. Setup then shows an approval link and a verification code.
Open the link, check that the page shows the same code, choose the account that will manage
the connector, and confirm. Setup pairs the connector and starts it with automatic updates.

For unattended setup, set each value in the environment instead, for example
`MIRAGE_PROXMOX_URL`, `MIRAGE_PROXMOX_TOKEN_ID`, `MIRAGE_PROXMOX_TOKEN_SECRET`,
`MIRAGE_PROXMOX_TEMPLATE_NODE`, and `MIRAGE_PROXMOX_TEMPLATE_VMID`. Setup fails early and
names any missing or malformed value.

## Validate and operate

Linked clones are the default. Set `MIRAGE_PROXMOX_FULL_CLONE=true` in
`/etc/mirage/agent.env` only when your storage or node topology requires it and measured
clone-to-ready time still fits Mirage’s 60-second startup window. After changing the
configuration, rerun the checks and restart the connector:

Connector host · check and restart:

```bash
sudo -u mirage bash -c \
  'set -a; source /etc/mirage/agent.env; mirage-linux-agent doctor'
sudo systemctl restart mirage-linux-agent
sudo journalctl -u mirage-linux-agent -f
```

The connector cleans up interrupted Mirage clones before it advertises capacity. It refuses
to delete a VM unless the VMID is in its reserved range and the VM also has Mirage’s exact
name and tag ownership markers.

If an unmanaged VM occupies a reserved VMID, or a disposable VM cannot be safely removed,
the connector fails closed and advertises no capacity. Resolve the conflict or cleanup
failure, then restart it. Use the machine page to confirm the Linux profile reports
**Ready** before routing jobs to it.

## Retained Linux VMs

Retained VM support is opt-in after the server supports `agent-vm-commands-v1`. Update
Mirage CLI to 0.2.34 or newer and the Linux connector to 0.4.4 or newer. Configure
`MIRAGE_PROXMOX_AGENT_VMID_BASE` and optionally `MIRAGE_PROXMOX_AGENT_VMID_COUNT` (default
1024) in the connector environment. This range must be separate from CI’s VMID range and
exclude the template. Never move a range while it contains retained guests.

Use an operator-prepared template containing `/usr/local/bin/mirage-agent-vm` from the
Mirage runner image, OpenSSH, the `runner` user, SFTP, Docker, Python 3 and the updated
Proxmox guest handoff service. Add `mirage-agent-vm-v1` to its description and set its
native architecture explicitly. Remove `/etc/mirage-agent-vm.json` and guest SSH host keys
before sealing a template. Keep QEMU Guest Agent enabled and disable Start at boot and
Protection. The connector must reach the guests’ private network on port 22. Use one connector per target node. Set `MIRAGE_RUNNER_ARCH` to `x64` or `arm64` for the template’s architecture, even when it differs from the appliance. Set `MIRAGE_RUNNER_ROOT_DISK_GIB` to the template’s scsi0 or virtio0 root disk size so admission charges its full cloned disk capacity.

The API token needs propagated `VM.Audit` on `/vms` for complete external inventory,
`Sys.Audit` on the target node, storage audit access and the existing Mirage pool lifecycle
permissions. These audit grants expose metadata without granting lifecycle control over
external guests. Incomplete visibility is reported and blocks admission. The resource
budget reads the target node and the template’s storage, not the connector appliance.
The target's status API must report `memory.available` (the kernel's MemAvailable).
Admission checks both configured VM allocations and measured memory availability
after the host reserve. If availability falls below that reserve, active Mirage
guests stop: CI disks are cleaned up and retained VM disks stay intact. Pressure and
compression counters remain unavailable when the API does not provide them.
Node architecture and template architecture must match the requested native profile.
Arm64 additionally requires compatible Proxmox hardware.

Readiness boots a temporary independent full clone, verifies the guest SSH service and
connector connectivity, and removes that probe. Retained guests always use full clones,
even if disposable CI guests use linked clones. The connector persists UUID-to-VMID
ownership mappings in its private state directory and validates them before every action.
Restart reconciliation stops retained runtimes and keeps their disks. CI cleanup operates
only in its separate range. Back up the connector state with its retained disks; losing
those mappings requires operator recovery and does not authorize adopting or deleting VMs.