Proxmox runners

When to use Proxmox

Choose Proxmox when you operate a Proxmox VE cluster and want a QEMU VM boundary for each Linux job. The connector is a lightweight service that talks to Mirage and the Proxmox HTTPS API; it does not need Docker, and its own CPU and memory do not limit how many jobs it runs. Run it in one of two places:

• On the Proxmox node. On a single node, install it directly on the host. No always-on VM is needed.
• On a small connector VM or machine. When it is a Proxmox VM, enable Start at boot for it so the connector returns after a node reboot. Only the connector’s own VM gets Start at boot; the template and every job clone keep it off.

Use one connector for each template architecture and target node. Workloads request their own sizes; older connectors use the same source template, target node, CPU count, memory, and disk contract. The connector supports one template and one target node; it does not provide HA placement, multi-node capacity discovery, automatic template creation, or a warm VM pool.

Prepare a VM template

Build a Linux x64 or arm64 VM with systemd that has:

• the same runner version and software payload as the Mirage runner image, including Docker Engine and /usr/local/bin/mirage-entrypoint;
• qemu-guest-agent installed;
• outbound HTTPS and GitHub Actions connectivity;
• no persistent job workspace, repository credential, or copied production SSH host key;
• a root disk at least as large as the connector’s runner disk setting.

Inside the VM, as a user with sudo, run:

bash <(curl -fsSL https://mirageci.com/bootstrap/proxmox-template)

Setup checks the VM, verifies a signed Mirage release, and installs the one-time configuration handoff: a small service that reads a job’s runner configuration from tmpfs, deletes it before starting the runner, and reports only readiness and the final exit code to the connector. It disables the guest’s own docker.service and docker.socket, because the runner starts an isolated Docker daemon, and enables the guest agent and the handoff. It also empties /etc/machine-id, so each clone generates its own. Running it again is safe.

Then finish the template in Proxmox:

1. Shut the VM down, and don’t start it again before converting it. A boot gives the VM a machine ID that every clone then shares, and clones that run at once are handed the same DHCP address. If it does start, run setup again before shutting it down.
2. In its Options, enable QEMU Guest Agent and disable Start at boot and Protection, so interrupted clones stay safe to destroy.
3. Add the tag mirage-template, then convert the VM to a template.
4. Note the template’s node and VMID for the connector.

Grant Proxmox access

Create a dedicated Proxmox API token and, preferably, a dedicated mirage pool. Scope the token to audit the source template and VM inventory; clone and configure VMs in the target pool and storage; use the selected network; start, stop, and delete those VMs; and call the QEMU Guest Agent. Exact ACL paths depend on your pool, storage, and SDN layout. Do not use an administrator token.

On Proxmox VE 9, this grant works for one node with a mirage pool, job disks on local-zfs, and a plain Linux bridge (the built-in localnetwork zone). Substitute your node, storage, zone, and template VMID:

pveum pool add mirage
pvesh set /pools/mirage --vms <template-vmid>
pveum user add mirage@pve --comment "Mirage Proxmox connector"
pveum user token add mirage@pve connector --privsep 1  # prints the secret once

pveum role add MirageRunnerConnector --privs "Pool.Audit,VM.Allocate,VM.Audit,\
VM.Clone,VM.Config.CPU,VM.Config.Memory,VM.Config.Options,VM.GuestAgent.Audit,\
VM.GuestAgent.FileRead,VM.GuestAgent.FileWrite,VM.PowerMgmt"
pveum role add MirageTaskAudit --privs Sys.Audit
pveum role add MirageSDNUse --privs SDN.Use

# A privilege-separated token gets the intersection of user and token ACLs.
for who in "--users mirage@pve" "--tokens mirage@pve!connector"; do
  pveum acl modify /pool/mirage --roles MirageRunnerConnector $who
  pveum acl modify /nodes/<node> --roles MirageTaskAudit --propagate 0 $who
  pveum acl modify /sdn/zones/localnetwork --roles MirageSDNUse $who
  pveum acl modify /storage/local-zfs --roles PVEDatastoreUser $who
done

Proxmox VE 8 has no VM.GuestAgent.* privileges; grant VM.Monitor there instead. Keep the token ID (mirage@pve!connector) and its secret for the next step. TLS certificate verification cannot be disabled, so if Proxmox uses a private certificate authority, keep its PEM certificate too.

Set up the connector

On the Proxmox node, or on a dedicated Linux x64 or arm64 host with systemd and HTTPS access to mirageci.com and your Proxmox API, run as a user with sudo, or as root on a node without it:

bash <(curl -fsSL https://mirageci.com/bootstrap/linux) --proxmox

Setup installs the signed Mirage agent, then asks for:

On the node itself, use https://127.0.0.1:8006 as the API URL and /etc/pve/pve-root-ca.pem as the CA certificate; the node certificate covers the local address. The token secret is read without echo and stored only in /etc/mirage/proxmox-token, readable by the Mirage service account alone. The reserved VMID range starts at the first VMID and holds one ID per concurrent job; keep the template outside it.

Before pairing, setup runs the connector’s checks: API access, the template flag and tag, the QEMU Guest Agent option, and the reserved VMID range. If a check fails, fix the value in /etc/mirage/agent.env (with sudoedit, or directly as root) and run setup again; the host does not appear in Mirage until the checks pass. Setup then shows an approval link and a verification code. Open the link, check that the page shows the same code, choose the account that will manage the connector, and confirm. Setup pairs the connector and starts it with automatic updates.

For unattended setup, set each value in the environment instead, for example MIRAGE_PROXMOX_URL, MIRAGE_PROXMOX_TOKEN_ID, MIRAGE_PROXMOX_TOKEN_SECRET, MIRAGE_PROXMOX_TEMPLATE_NODE, and MIRAGE_PROXMOX_TEMPLATE_VMID. Setup fails early and names any missing or malformed value.

Validate and operate

Linked clones are the default. Set MIRAGE_PROXMOX_FULL_CLONE=true in /etc/mirage/agent.env only when your storage or node topology requires it and measured clone-to-ready time still fits Mirage’s 60-second startup window. After changing the configuration, rerun the checks and restart the connector:

sudo -u mirage bash -c \
  'set -a; source /etc/mirage/agent.env; mirage-linux-agent doctor'
sudo systemctl restart mirage-linux-agent
sudo journalctl -u mirage-linux-agent -f

The connector cleans up interrupted Mirage clones before it advertises capacity. It refuses to delete a VM unless the VMID is in its reserved range and the VM also has Mirage’s exact name and tag ownership markers.

If an unmanaged VM occupies a reserved VMID, or a disposable VM cannot be safely removed, the connector fails closed and advertises no capacity. Resolve the conflict or cleanup failure, then restart it. Use the machine page to confirm the Linux profile reports Ready before routing jobs to it.

Retained Linux VMs

Retained VM support is opt-in after the server supports agent-vm-commands-v1. Update Mirage CLI to 0.2.34 or newer and the Linux connector to 0.4.4 or newer. Configure MIRAGE_PROXMOX_AGENT_VMID_BASE and optionally MIRAGE_PROXMOX_AGENT_VMID_COUNT (default 1024) in the connector environment. This range must be separate from CI’s VMID range and exclude the template. Never move a range while it contains retained guests.

Use an operator-prepared template containing /usr/local/bin/mirage-agent-vm from the Mirage runner image, OpenSSH, the runner user, SFTP, Docker, Python 3 and the updated Proxmox guest handoff service. Add mirage-agent-vm-v1 to its description and set its native architecture explicitly. Remove /etc/mirage-agent-vm.json and guest SSH host keys before sealing a template. Keep QEMU Guest Agent enabled and disable Start at boot and Protection. The connector must reach the guests’ private network on port 22. Use one connector per target node. Set MIRAGE_RUNNER_ARCH to x64 or arm64 for the template’s architecture, even when it differs from the appliance. Set MIRAGE_RUNNER_ROOT_DISK_GIB to the template’s scsi0 or virtio0 root disk size so admission charges its full cloned disk capacity.

The API token needs propagated VM.Audit on /vms for complete external inventory, Sys.Audit on the target node, storage audit access and the existing Mirage pool lifecycle permissions. These audit grants expose metadata without granting lifecycle control over external guests. Incomplete visibility is reported and blocks admission. The resource budget reads the target node and the template’s storage, not the connector appliance. The target's status API must report memory.available (the kernel's MemAvailable). Admission checks both configured VM allocations and measured memory availability after the host reserve. If availability falls below that reserve, active Mirage guests stop: CI disks are cleaned up and retained VM disks stay intact. Pressure and compression counters remain unavailable when the API does not provide them. Node architecture and template architecture must match the requested native profile. Arm64 additionally requires compatible Proxmox hardware.

Readiness boots a temporary independent full clone, verifies the guest SSH service and connector connectivity, and removes that probe. Retained guests always use full clones, even if disposable CI guests use linked clones. The connector persists UUID-to-VMID ownership mappings in its private state directory and validates them before every action. Restart reconciliation stops retained runtimes and keeps their disks. CI cleanup operates only in its separate range. Back up the connector state with its retained disks; losing those mappings requires operator recovery and does not authorize adopting or deleting VMs.