vfio-isolate is a command line tool for Linux that facilitates CPU and memory isolation for running virtual machines with guaranteed latency. It partitions CPUs between the host and a VM (cpusets), moves IRQs, unbound workqueues and kernel threads away from the VM's CPUs, sets the CPU governor, drops caches, compacts memory and reserves hugepages, and records how to undo all of it.
For the full option list of any command run vfio-isolate --help or vfio-isolate <command> --help.
- Linux with cgroup v2 (the unified hierarchy) and the
cpusetcontroller available. cgroup v1 is not supported since 1.0; v1 users stay on 0.5.2:pip install vfio-isolate==0.5.2. - Python 3.9 or newer.
- Root. Every run that queues a command, and
restore, is refused withvfio-isolate: must be run as root(exit status 1) for other users.--helpand usage errors do not need root. - systemd is optional. On systemd hosts, cpusets of systemd units are changed through systemd (see cpuset-modify).
pip install vfio-isolateUpgrading from 0.5.x: see CHANGELOG.md for the breaking changes.
Arguments that take CPUs accept three forms:
C<list>: decimal CPU numbers and ranges, e.g.C0-3,8.Cis never read as a mask.N<list>: NUMA nodes, expanded to all CPUs of those nodes, e.g.N0.M<mask>: a hex CPU mask as in/proc/irq/*/smp_affinity: groups of 1 to 8 hex digits separated by commas, most significant group first, each group 32 bits, case-insensitive.M3cis CPUs 2-5;Mff,0000000fis CPUs 0-3 and 32-39.
Arguments that take NUMA memory nodes (--mems) accept only N<list>, e.g. N0 or N0-1.
vfio-isolate irq-affinity mask C2-5,8-11
vfio-isolate irq-affinity mask N0
vfio-isolate irq-affinity mask Mff00Syntax errors, reversed ranges (C7-0), ranges ending above 65535, empty sets (C, N, M, M0), and CPUs or NUMA
nodes that do not exist (including N nodes without CPUs given as a CPU argument) are usage errors (exit status 2).
They are reported before anything is changed.
vfio-isolate does not pick CPUs for you. A common split:
- Give the VM whole physical cores (both SMT threads of each), ideally a whole L3 cache domain (a CCD or CCX on AMD).
- Keep CPU 0 and its SMT sibling for the host.
- On NUMA hosts keep the VM's CPUs and its memory on one node.
Read the topology from sysfs:
# SMT siblings (one line per CPU)
grep . /sys/devices/system/cpu/cpu*/topology/thread_siblings_list
# L3 cache domains: CPUs sharing an L3, and its size (index3 is usually L3, so check `level`)
for c in /sys/devices/system/cpu/cpu[0-9]*/cache/index*; do
[ "$(cat $c/level)" = 3 ] && echo "$(cat $c/shared_cpu_list) $(cat $c/size)"
done | sort -u
# Intel hybrid CPUs only (these files do not exist elsewhere)
cat /sys/devices/cpu_core/cpus /sys/devices/cpu_atom/cpus
# NUMA nodes
grep . /sys/devices/system/node/node*/cpulistlscpu -e and lstopo (from hwloc) show the same information.
On AMD X3D CPUs with two CCDs, the CCD with the larger L3 is the V-cache CCD. Which side should get it depends on the workload; giving it to the VM is a common choice for games. On Intel hybrid CPUs, giving the P-cores to the VM is the common choice. This project has not tested X3D or hybrid hardware; treat both as reports from users, not as recommendations.
Example: a 16-core/32-thread CPU with two CCDs, CPUs 0-7,16-23 on CCD0 and 8-15,24-31 on CCD1. The host keeps CCD0
(C0-7,16-23), the VM gets CCD1 (C8-15,24-31):
sudo vfio-isolate -u /run/vfio-isolate.undo \
cpuset-modify --cpus C0-7,16-23 /system.slice \
cpuset-modify --cpus C0-7,16-23 /user.slice \
irq-affinity mask C8-15,24-31 \
workqueue-affinity mask C8-15,24-31 \
kthread-affinity mask C8-15,24-31Commands can be chained in one invocation. They run in the order given, and all arguments are validated before the first command runs.
sudo vfio-isolate \
drop-caches \
cpu-governor performance C8-15,24-31 \
compact-memorycpu-governor additionally checks, while arguments are parsed, that every selected CPU has a cpufreq directory and
offers the governor; otherwise the whole run is a usage error and nothing is executed. The kernel loads a governor
module on demand when its name is written to scaling_governor, but scaling_available_governors lists only loaded
governors, so such a governor is refused: load it first (modprobe cpufreq_<name>).
With -u <file> vfio-isolate records how to undo each command in a journal file (plain text, JSON Lines).
sudo vfio-isolate -u /run/vfio-isolate.undo cpuset-create --cpus C1-4 /vfio-test
sudo vfio-isolate restore /run/vfio-isolate.undo- The undo steps of each command are appended to the file and made durable before that command changes anything, so the journal is complete even if a run fails half way. Commands that never started record nothing.
- If the file already exists and is a journal, later runs with
-uappend to it.restorethen undoes every recorded run, newest first, back to the state before the first one. restorecontinues past steps that fail and reports them on stderr. If everything succeeded the file is deleted; otherwise it is rewritten to contain only the failed steps, and the exit status is 1, so you can fix the problem and runrestoreagain.- The file is created with mode 0600. Symlinks, files not owned by the current user and files writable by group or
others are refused for both
-uandrestore, as is an existing-ufile that is not a journal. These checks happen before any command of the chain makes a change.-uandrestorecannot be used in the same command. - The journal only describes known commands and their parameters; restoring it does not execute arbitrary code.
- Undo files written by vfio-isolate 0.5.x are not supported and are refused. Run
restorewith 0.5.2 (pip install vfio-isolate==0.5.2) for any VM prepared with 0.5.x before upgrading. - Keep the undo file on a volatile path such as
/run: cpuset state does not survive a reboot either.
-n (--dry-run) checks a chain and prints what it would do, without changing anything: no write under /sys,
/proc or the cgroup tree, no state-changing systemctl call, no journal and no lock. Use it to try a hook line
before the first real run.
vfio-isolate -n -u /run/vfio-isolate.undo \
cpuset-modify --cpus C0-7,16-23 /system.slice \
irq-affinity mask C8-15 \
workqueue-affinity mask C8-15 \
hugepages 2M 16G \
cpu-governor performance C8-15would record undo in /run/vfio-isolate.undo
would set unit system.slice (systemd): AllowedCPUs 0-31 -> 0-7,16-23
would set default IRQ affinity: 0-31 -> 0-7,16-31
would change affinity of 41 of 57 IRQs (mask 8-15)
would set unbound workqueue mask: 0-31 -> 0-7,16-31
would add 16G in 2M pages (8192) to the pool, now 0; whether the kernel can supply them shows only in a real run
would set governor of CPUs 8-15: schedutil -> performance
- Preconditions are checked in order, as in a real run. The first one that fails ends the preview with
vfio-isolate: dry run: <command> failed: <reason>and exit status 1. - The preview follows the chain: a cpuset created by an earlier step exists for the later ones (
set cpuset /a (new): cpus 8-15), and task counts that depend on earlier steps are marked(after earlier steps). Other values an earlier step changes (a cpuset's CPUs, a governor, an IRQ's affinity, the workqueue mask, a kernel thread's affinity, the hugepage pool) are not simulated: the later step shows(after earlier steps)instead of the current value, and a precondition that depends on them is not evaluated; the preview says(precondition checked only in a real run: depends on earlier steps). - The CPUs that earlier steps put into isolated cpuset partitions are tracked where they are known exactly (a
cpuset-modify --cpus ... --partition isolatedon a cpuset that is not a partition yet, including one created earlier in the chain). A laterworkqueue-affinityorkthread-affinitymask/addthen refuses exactly like a real run (exit status 1). Where the isolation is not exactly known (a partition without--cpus, new CPUs for a partition or for the parent of one, deleting a partition, a restore step that changes one) the preview shows the note above instead. - With
-v,irq-affinityalso lists each IRQ that would change (IRQ 24: 0-31 -> 0-7,16-23) andkthread-affinityeach kernel thread (2 kthreadd: 0-31 -> 0-7,16-31). Kernel threads that refuse anyway (a race) are only known in a real run. restore -n FILE(or-n restore FILE; give the option before FILE) reads and validates the journal and prints the undo steps, newest record first; the file is not touched.restore FILE -n(option after FILE) is a usage error. Restore steps are not preconditions-checked, as in a real restore.- With
-n -u FILE, FILE is validated as a real run would: header, 0.5.x format, records, mode, and its owner, which must be root because a real run is (a file owned by you is refused). A truncated last line is reported (would drop truncated last line of FILE), not trimmed. A directory that does not exist fails as in a real run. If you cannot read an existing journal (root-owned, mode 0600), the preview sayswould record undo in FILE (existing journal not readable as this user; checked only in a real run).
vfio-isolate restore -n /run/vfio-isolate.undo- A dry run does not need root where the files it reads are readable by the current user; a value it cannot read is
shown as
unknown (permission denied)(ordoes not exist, or the error text). Journals are root-owned and mode 0600, sorestore -nnormally needs root. - A dry run cannot know what only the kernel decides: whether it can supply the hugepages, which IRQs refuse the new
affinity (
EIO, orEPERMon newer kernels), which tasks cannot be moved, and whether a cpuset partition is accepted.
- IRQs and tasks that cannot be changed do not stop a run.
irq-affinityandmove-tasks(and their undo duringrestore) count expected failures per kind and print one summary line per kind at the end: kernel-managed IRQs cannot be moved, vanished IRQs, kernel threads and exited tasks. With-veach item is listed. Any other error is printed as a warning naming the item the first time it occurs; further items with the same error are only counted. - If one
irq-affinityormove-taskscommand attempted at least one item, changed none, and at least one failure was unexpected, the run continues but ends with status 1 and the linevfio-isolate: <command>: no IRQ could be changed(orno task could be moved). - A command that fails while it is checked, recorded or executed stops the run with
vfio-isolate: <command> failed: <reason>. The message tells you how to get back: with-uand at least one undo record written, thatvfio-isolate restore <file>undoes the changes made so far; without-u, once a command has started executing, that changes made before the failure remain. - An unexpected exception prints one line with
use -d for a traceback;-dshows the traceback. -venables verbose output,-ddebug output.
| Status | Meaning |
|---|---|
| 0 | success |
| 1 | failure, including a run where irq-affinity or move-tasks changed nothing because of unexpected errors; also a restore that kept failed steps |
| 2 | usage error |
| 130 | interrupted (Ctrl-C); before the first change it reports that nothing was changed; with -u and records written it prints the restore hint; an interrupted restore leaves its file for another restore |
All commands are given after the global options -v, -d, -n and -u <undo-file>.
vfio-isolate compact-memory
Asks the kernel to compact memory (writes 1 to /proc/sys/vm/compact_memory), which makes huge pages easier to
allocate. Takes no arguments. Nothing to undo.
vfio-isolate cpu-governor GOVERNOR CPUS
Sets the CPU frequency governor of the given CPUs (C, N or M form). The governor is checked against each CPU's
available governors before anything runs (a governor module that is not loaded is refused; load it first with
modprobe cpufreq_<name>). Undo restores each CPU's previous governor.
sudo vfio-isolate -u /run/vfio-isolate.undo cpu-governor performance C2-5,8-11vfio-isolate cpuset-create [--cpus CPUS] [--mems NODES] NAME
Creates the cpuset (cgroup) NAME, e.g. /vfio-test.
--cpus CPUS: CPUs of the cpuset (C,NorMform).--mems NODES: NUMA memory nodes (Nform).- Without
--cpus/--memsthe new cpuset inherits its parent's values. - Refused for systemd units and
/, and below a systemd unit that does not delegate its subtree (Delegate=yes); these checks happen before any change. Thecpusetcontroller is enabled in the ancestors as needed. - Undo deletes the cpuset.
sudo vfio-isolate -u /run/vfio-isolate.undo cpuset-create --cpus C1-4 --mems N0 /vfio-testvfio-isolate cpuset-delete NAME
Moves all processes of the cpuset to its parent cpuset, then removes it. Refused for systemd units and /. Undo
recreates the cpuset with its previous cpus and mems; the processes are not moved back.
A cpuset that has child cgroups is refused; delete the children first. A cpuset that is briefly busy (new processes appeared after the move) is retried for about 2 seconds, moving the tasks again, before failing with "still in use".
sudo vfio-isolate -u /run/vfio-isolate.undo cpuset-delete /vfio-testvfio-isolate cpuset-modify [--cpus CPUS] [--mems NODES] [--partition member|root|isolated] NAME
Changes the CPUs, memory nodes and (experimental) partition type of an existing cpuset. Refused for /.
--cpus CPUS: CPUs (C,NorMform).--mems NODES: NUMA memory nodes (Nform).--partition member|root|isolated: EXPERIMENTAL, sets the cgroup v2 partition type; see Exclusive CPUs with a partition.- Systemd units (
*.slice,*.scope,*.service, ...): when systemd reports the unit's cgroup asNAME, the options are applied as runtime unit properties withsystemctl set-property --runtime(AllowedCPUs=,AllowedMemoryNodes=,CPUSetPartition=) instead of writing cgroup files, so systemd stays the owner of the cgroup. The setting survivessystemctl daemon-reload; a value written directly intocpuset.cpusof a slice would be reset by a reload. If the unit name and systemd'sControlGroupdisagree, the command fails before any change. - Other cgroups, and hosts without systemd: written directly, after enabling the
cpusetcontroller in the ancestors. - Undo restores the previous configured values (an empty value, meaning inherit from the parent, is restored as
empty; on the systemd route the previous properties are restored). For a partition, undo restores the previous partition type (on the systemd route a previously unset value is restored as
member; empty is never restored).
sudo vfio-isolate -u /run/vfio-isolate.undo cpuset-modify --cpus C0-6,16-22 /system.slicevfio-isolate drop-caches
Drops the page cache, dentries and inodes (writes 3 to /proc/sys/vm/drop_caches). Takes no arguments. Nothing to
undo.
vfio-isolate hugepages [--node N<n>] PAGESIZE AMOUNT
Adds AMOUNT of memory in pages of PAGESIZE to the hugepage pool (nr_hugepages); the amount is relative to the
current pool size, not a target. Undo releases the same number of pages again, never below zero, whatever else changed the pool meanwhile, so
overlapping VMs that each add and release their own pages are fine. Pages that are still in use are kept by the kernel
until their user exits, so a restore while the VM still runs does not break it; they return to the free pool when the
VM is gone.
PAGESIZE: page size as2M,1Gor2048k. It must match the page size of the VM (libvirt:<page size=...>in<memoryBacking>). Sizes the kernel does not offer are usage errors.AMOUNT: memory to add, as16G,512Mand so on; it must be an exact multiple of the page size (hugepages 2M 16Gadds 8192 pages).--node N<n>: add the pages on one NUMA node instead of system-wide, for exampleN0. Exactly one node, and it must have a pool of that size.- If the kernel provides fewer pages than asked, the command fails with
got 6G of 16G in 1G pages, releases what it got and leaves no undo record, so arestoreof the same undo file afterwards is safe. - Runs of vfio-isolate are serialized by a lock (
/run/vfio-isolate.lock), so two hooks starting at the same time do not race on the pool. - To use the pages, QEMU needs hugepage backing; for libvirt add
<memoryBacking><hugepages><page size='2048' unit='KiB'/></hugepages></memoryBacking>to the domain XML. Without hugepage backing, the pool stays unused and only takes memory away from the host.
sudo vfio-isolate -u /run/vfio-isolate.undo hugepages 2M 16GUse 2M pages with this command. They allocate at runtime, are taken only while the VM runs and go back to the
host when it stops. Tested on a real host with GPU and NVMe passthrough (hugepages 2M 16G, all 8192 pages in use
by the VM, started shortly after boot) and in a VM with heavily fragmented memory.
1G pages need physically contiguous gigabytes, and at runtime they usually fail:
- On a desktop host after 11 hours of uptime,
hugepages 1G 8Ggot 0 of 8 pages with 31 GB free, with and withoutcompact-memoryfirst. Unmovable and reclaimable kernel memory is spread over all of RAM, so no free gigabyte is left. - What works is reserving them at boot with
hugepagesz=1G hugepages=Non the kernel command line, without this command. That memory is gone for the host even while the VM is off.
The gain of 1G over 2M pages is one level less of address translation; for a gaming VM it is likely small, and with transparent hugepages (on by default) the VM already gets 2M pages for much of its memory.
vfio-isolate irq-affinity mask|add CPUS
Changes the affinity of all active IRQs. It first applies the same operation to /proc/irq/default_smp_affinity,
the affinity every IRQ gets when it is created.
- The default matters because IRQs created later start with it: QEMU requests the VFIO IRQs after the libvirt
preparehook has run, so without it they land on the VM's CPUs. - A
maskthat would leave no online CPU in the default is refused before anything changes (exit status 1). mask CPUS: removes the CPUs from each IRQ's affinity. If none would remain, the IRQ gets all other CPUs.add CPUS: adds the CPUs to each IRQ's affinity.- Kernel-managed IRQs cannot be moved; they keep their affinity and are summarized (see Error reporting).
- Undo restores each IRQ's previous affinity. For the default it is relative: it gives back only the CPUs this run removed (or removes only the CPUs it added), so overlapping VMs do not hand each other's CPUs back.
- IRQs created while the VM runs are not journaled: host IRQs created in that window (hotplug, driver reload) keep the restricted affinity after the restore. VFIO IRQs disappear with QEMU.
irqbalancerewrites IRQ affinities periodically and so silently undoes this command. If a process namedirqbalanceis running, the command prints one warning to stderr per invocation (and still runs). Either stop it around the VM (systemctl stop irqbalance, start it again afterwards) or ban the VM's CPUs in its configuration withIRQBALANCE_BANNED_CPULIST. The config file differs by distribution, e.g./etc/default/irqbalanceor/etc/sysconfig/irqbalance. irqbalance automatically avoids CPUs isolated with the kernel parametersisolcpus=andnohz_full=(it reads/sys/devices/system/cpu/isolatedandnohz_full), but most likely not CPUs isolated by cpuset partitions: with partitions, setIRQBALANCE_BANNED_CPULISTor stop irqbalance.IRQBALANCE_BANNED_CPUS(a hex mask) is deprecated since irqbalance 1.8.0.
sudo vfio-isolate -u /run/vfio-isolate.undo irq-affinity mask C2-5,8-11
sudo vfio-isolate irq-affinity add C2-5,8-11vfio-isolate kthread-affinity mask|add CPUS
Keeps kernel threads (kswapd, khugepaged, kcompactd, jbd2/..., ...) off the VM's CPUs. Kernel threads stay in
the root cgroup, so cpusets and move-tasks cannot do it. The command changes the CPU affinity of kthreadd
(PID 2), whose affinity every new kernel thread inherits, and of every kernel thread that accepts a new affinity. It
replaces the taskset -pc ... 2 workaround of older hook scripts, which had no undo and left running kernel threads
where they were.
mask CPUS: removes the CPUs from each thread's affinity; a thread that would have none left gets all other online CPUs.add CPUS: adds the CPUs to each thread's affinity.- The kernel threads are listed when the command line is parsed (threads with
PF_KTHREADin/proc/<pid>/stat); threads created later inherit whatkthreaddhas. - Per-CPU kernel threads (
ksoftirqd/N,kworker/N:...,migration/N, ...;PF_NO_SETAFFINITY) always refuse a new affinity and are skipped, so they appear neither in the output nor in the undo file. A thread that still refuses (EINVAL) or that exited meanwhile (ESRCH) is only counted (vfio-isolate: 1 kernel thread exited before it could be moved; with-veach thread is listed). Any other error is reported once; the command fails only if no thread could be changed because of such an unexpected error. - Refused while CPUs are in isolated cpuset partitions: the kernel then restricts
kthreadditself, and an affinity recorded with a partition active would keep kernel threads off those CPUs after the partition is gone. If combined, put it before the partition step (kthread-affinity mask ... cpuset-modify --partition isolated ...). - Undo restores each thread's previous affinity. A thread that exited meanwhile, or whose PID now belongs to another
process or kernel thread (checked by the thread's start time), is left alone and counts as restored. Restore does
not re-check isolation: if a different isolated partition is active at release,
kthreadd's previous affinity is restored as recorded (accepted by design).
sudo vfio-isolate -u /run/vfio-isolate.undo kthread-affinity mask C8-15,24-31
sudo vfio-isolate kthread-affinity add C8-15,24-31vfio-isolate move-tasks FROM TO
Moves all tasks from cpuset FROM to cpuset TO. Kernel threads and tasks that exited meanwhile are skipped and
summarized. Undo moves back exactly the processes that were in FROM when the command ran and are still alive and in
TO; processes that were already in TO stay. On systemd hosts prefer restricting slices with cpuset-modify over
moving processes out of their units, which confuses systemd's accounting.
sudo vfio-isolate -u /run/vfio-isolate.undo move-tasks /vfio-test /vfio-isolate workqueue-affinity mask|add CPUS
Restricts the CPUs that unbound kernel workqueues (writeback, block I/O, crypto, ...) run on, by
changing /sys/devices/virtual/workqueue/cpumask. Per-CPU (bound) kworkers (kworker/N:...) cannot be moved and
are not affected.
mask CPUS: removes the CPUs from the mask.add CPUS: adds the CPUs to the mask.- The new mask is computed from the last requested mask. A result without any CPU is refused.
- Refused while CPUs are in isolated cpuset partitions: the kernel already keeps unbound workqueues off them, and
writing the mask would lose them from it when the partition is removed. It is not needed with isolated partitions;
if combined, put it before the partition step (
workqueue-affinity mask ... cpuset-modify --partition isolated ...). - A result without an online CPU is refused.
- Undo restores the previous requested mask, minus the CPUs isolated at that time.
sudo vfio-isolate -u /run/vfio-isolate.undo workqueue-affinity mask C2-5,8-11
sudo vfio-isolate workqueue-affinity add C2-5,8-11vfio-isolate restore UNDO_FILE
Undoes the runs recorded in the journal, newest first. Continues past failing steps; deletes the file on success,
otherwise keeps only the failed steps and exits with status 1. Cannot be combined with -u. See
Undo journal.
sudo vfio-isolate restore /run/vfio-isolate.undoIn this example, an AMD 5950x is partitioned between host and VM. It assumes systemd, which has already created the
cgroups system.slice and user.slice.
The VM exclusively uses the 8 cores of the second die. The last core of the first die is used for emulation and IO work, and the first seven are for the host.
sudo vfio-isolate \
cpuset-modify --cpus C0-6,16-22 /system.slice \
cpuset-modify --cpus C0-6,16-22 /user.sliceThis instructs the two existing cgroups that systemd created to only use the first 7 cores (and their SMT siblings). To pin the VM cores, use libvirt:
<vcpu placement='static'>16</vcpu>
<iothreads>1</iothreads>
<cputune>
<vcpupin vcpu='0' cpuset='8'/>
<vcpupin vcpu='1' cpuset='24'/>
<vcpupin vcpu='2' cpuset='9'/>
<vcpupin vcpu='3' cpuset='25'/>
<vcpupin vcpu='4' cpuset='10'/>
<vcpupin vcpu='5' cpuset='26'/>
<vcpupin vcpu='6' cpuset='11'/>
<vcpupin vcpu='7' cpuset='27'/>
<vcpupin vcpu='8' cpuset='12'/>
<vcpupin vcpu='9' cpuset='28'/>
<vcpupin vcpu='10' cpuset='13'/>
<vcpupin vcpu='11' cpuset='29'/>
<vcpupin vcpu='12' cpuset='14'/>
<vcpupin vcpu='13' cpuset='30'/>
<vcpupin vcpu='14' cpuset='15'/>
<vcpupin vcpu='15' cpuset='31'/>
<emulatorpin cpuset='7,23'/>
<iothreadpin iothread='1' cpuset='7,23'/>
<vcpusched vcpus='0' scheduler='rr' priority='1'/>
<vcpusched vcpus='1' scheduler='rr' priority='1'/>
<vcpusched vcpus='2' scheduler='rr' priority='1'/>
<vcpusched vcpus='3' scheduler='rr' priority='1'/>
<vcpusched vcpus='4' scheduler='rr' priority='1'/>
<vcpusched vcpus='5' scheduler='rr' priority='1'/>
<vcpusched vcpus='6' scheduler='rr' priority='1'/>
<vcpusched vcpus='7' scheduler='rr' priority='1'/>
<vcpusched vcpus='8' scheduler='rr' priority='1'/>
<vcpusched vcpus='9' scheduler='rr' priority='1'/>
<vcpusched vcpus='10' scheduler='rr' priority='1'/>
<vcpusched vcpus='11' scheduler='rr' priority='1'/>
<vcpusched vcpus='12' scheduler='rr' priority='1'/>
<vcpusched vcpus='13' scheduler='rr' priority='1'/>
<vcpusched vcpus='14' scheduler='rr' priority='1'/>
<vcpusched vcpus='15' scheduler='rr' priority='1'/>
<iothreadsched iothreads='1' scheduler='fifo' priority='98'/>
</cputune>cpuset-modify --partition member|root|isolated sets the cgroup v2 partition type. An isolated partition has no
load balancing at all, not even among its own CPUs, so every thread running in it must be pinned (vCPUs, emulator and
iothreads). That is stronger than a root partition, which only removes its CPUs from the scheduler domains of
others. Example: isolate
the CPUs of the VM in machine.slice. The partition must contain every CPU the VM's threads are pinned to, including
emulatorpin and iothreadpin; the recipe's XML pins those to 7,23, so the partition is C7-15,23-31 and the host
slices get C0-6,16-22:
sudo vfio-isolate -u /run/vfio-isolate.undo \
cpuset-modify --cpus C0-6,16-22 /init.scope \
cpuset-modify --cpus C0-6,16-22 /system.slice \
cpuset-modify --cpus C0-6,16-22 /user.slice \
cpuset-modify --cpus C7-15,23-31 --partition isolated /machine.sliceSiblings lose the partition's CPUs from their effective set. On kernel 7.2 that also holds for a sibling with explicitly
configured overlapping CPUs (tested); older kernels may refuse the partition then, so restricting the other slices first
as shown is the safe order. The kernel validates the
partition; if it ends up invalid, the tool reports the reason and exits with status 1, and restore repairs it.
Undo restores the previous partition type (on the systemd route a previously unset value is restored as member, never empty). On
systemd units, root and isolated need --cpus unless the unit already has AllowedCPUs. If the installed systemd
does not support CPUSetPartition=, the command fails before any change.
Pin vCPUs only to CPUs inside the partition. Tested with a libvirt VM (kernel 7.2, systemd 262, the XML above, 16
vCPUs, GPU and NVMe passthrough) through the libvirt hook with this chain in place of its
cpuset-modify, workqueue-affinity and kthread-affinity lines:
- The partition was valid, each vCPU thread ran on its pinned CPU, and the host slices stayed on 0-6,16-22.
- The kernel itself kept unbound workqueues and
kthreaddoff the partition, soworkqueue-affinityandkthread-affinityare not needed with it. restoreat VM shutdown brought every value back.- Without load balancing, QEMU's emulator threads are not spread over the CPUs of
emulatorpin: with7,23, six of seven sat on CPU 7. Giveemulatorpinandiothreadpinone CPU each (for example7and23) so that they do not share one CPU while the other stays idle. - Whether the partition lowers latency compared with the plain recipe above has not been measured.
Fully featured example, /etc/libvirt/hooks/qemu (make it executable). The undo file lives in libvirt's runtime
directory (root only, gone on reboot). If the isolation fails in prepare, the hook restores what was already changed
and exits non-zero, so libvirt does not start the VM; libvirt still calls release afterwards, which only restores
when the undo file exists.
#!/bin/bash
# change to your libvirt domain name
VM=win11
[ "$1" = "$VM" ] || exit 0
HCPUS=0-6,16-22
MCPUS=8-15,24-31
UNDOFILE=/run/libvirt/qemu/vfio-isolate-$VM.undo
enable_isolation () {
# VM backed by hugepages? Add "hugepages 2M 16G \" as its own line after
# the "compact-memory \" line below.
vfio-isolate \
-u $UNDOFILE \
drop-caches \
cpu-governor performance C$MCPUS \
cpuset-modify --cpus C$HCPUS /system.slice \
cpuset-modify --cpus C$HCPUS /user.slice \
compact-memory \
irq-affinity mask C$MCPUS \
workqueue-affinity mask C$MCPUS \
kthread-affinity mask C$MCPUS
}
disable_isolation () {
if [ -e "$UNDOFILE" ]; then
vfio-isolate restore $UNDOFILE
fi
}
case "$2" in
"prepare")
if ! enable_isolation; then
if [ -e "$UNDOFILE" ]; then
vfio-isolate restore $UNDOFILE
fi
exit 1
fi
;;
"started")
;;
"release")
disable_isolation
;;
esacOn Ubuntu the libvirtd AppArmor profile may block the hook from running vfio-isolate. Allow it in
/etc/apparmor.d/local/usr.sbin.libvirtd with a line like the following (use the path printed by
command -v vfio-isolate), then reload the profile with sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.libvirtd:
/usr/local/bin/vfio-isolate rmix,
Reported by users (#4), not tested by this project.
pip install -e '.[test]' and pytest run the unit tests against a fake cgroup tree. The tests/vm suite runs the
real CLI against a real kernel and systemd inside a throwaway QEMU VM (the host kernel, root file system read-only
with an in-memory overlay). It needs virtme-ng (sudo pacman -S virtme-ng or the distribution equivalent) and
/dev/kvm:
scripts/vm-test --venv /path/to/venvUse a venv in which pip install -e '.[test]' has been done. The run takes about 10 seconds; extra arguments are
passed to pytest. Nothing is run as root on the host.