Documentation/driver-api/vfio.rst

2 VFIO - "Virtual Function I/O" [1]_
7 allotted.  This includes x86 hardware with AMD-Vi and Intel VT-d,
9 systems such as Freescale PAMU.  The VFIO driver is an IOMMU/device
11 a secure, IOMMU protected environment.  In other words, this allows
12 safe [2]_, non-privileged, userspace drivers.
19 bare-metal device drivers [3]_.
22 field, also benefit from low-overhead, direct device access from
23 userspace.  Examples include network adapters (often non-TCP/IP based)
27 which has no notion of IOMMU protection, limited interrupt support,
33 secure, more featureful userspace driver environment than UIO.
36 ---------------------------
41 by far the most critical aspect for maintaining a secure environment
42 as allowing a device read-write access to system memory imposes the
50 things like secure direct assignment of devices into virtual machines.
53 though.  Even when an IOMMU is capable of this, properties of devices,
54 interconnects, and IOMMU topologies can each reduce this isolation.
55 For instance, an individual device may be part of a larger multi-
56 function enclosure.  While the IOMMU may be able to distinguish
58 transactions between devices to reach the IOMMU.  Examples of this
59 could be anything from a multi-function PCI device with backdoors
60 between functions to a non-PCI-ACS (Access Control Services) capable
61 bridge allowing redirection without reaching the IOMMU.  Topology
62 can also play a factor in terms of hiding devices.  A PCIe-to-PCI
64 from the bridge itself.  Obviously IOMMU design plays a major factor
67 Therefore, while for the most part an IOMMU may have device level
69 IOMMU API therefore supports a notion of IOMMU groups.  A group is
74 ensure secure user access, it's not necessarily the preferred
91 $GROUP is the IOMMU group number of which the device is a member.
92 If the IOMMU group contains multiple devices, each will need to
96 group available, but not that particular device).  TBD - interface
102 previously opened container file.  If desired and if the IOMMU driver
103 supports sharing the IOMMU context between groups, multiple groups may
109 ioctls become available, enabling access to the VFIO IOMMU interfaces.
119 ------------------
126 This device is therefore in IOMMU group 26.  This device is on the
127 pci bus, therefore the user will make use of vfio-pci to manage the
130 	# modprobe vfio-pci
132 Binding this device to the vfio-pci driver creates the VFIO group
135 	$ lspci -n -s 0000:06:0d.0
138 	# echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
143 	$ ls -l /sys/bus/pci/devices/0000:06:0d.0/iommu_group/devices
145 	lrwxrwxrwx. 1 root root 0 Apr 23 16:13 0000:00:1e.0 ->
147 	lrwxrwxrwx. 1 root root 0 Apr 23 16:13 0000:06:0d.0 ->
149 	lrwxrwxrwx. 1 root root 0 Apr 23 16:13 0000:06:0d.1 ->
152 This device is behind a PCIe-to-PCI bridge [4]_, therefore we also
156 bind this device to the vfio-pci driver (vfio-pci does not currently
166 The user now has full access to all the devices and the iommu for this
183 		/* Doesn't support the IOMMU driver we want. */
197 	/* Enable the IOMMU model we want */
200 	/* Get addition IOMMU info */
243 ----------------------------
256 container and IOMMU backend interfaces.  The compatibility mode can
258 simply symlink'd to /dev/iommu.  Note that at the time of writing, the
270 ----------------
287 VFIO device cdev doesn't rely on VFIO group/container/iommu drivers.
294 vfio device cdev access is still bound by IOMMU group semantics, ie. there
302 -------------------
306 	$ ls /sys/bus/pci/devices/0000:6a:01.0/vfio-dev/
312 	$ ls -l /dev/vfio/devices/vfio0
313 	crw------- 1 root root 511, 0 Feb 16 01:22 /dev/vfio/devices/vfio0
314 	$ cat /sys/bus/pci/devices/0000:6a:01.0/vfio-dev/vfio0/dev
316 	$ ls -l /dev/char/511\:0
317 	lrwxrwxrwx 1 root root 21 Feb 16 01:22 /dev/char/511:0 -> ../vfio/devices/vfio0
353 	iommufd = open("/dev/iommu", O_RDWR);
374 -------------------------------------------------------------------------------
379 -------------------------------------------------------------------------------
381 VFIO bus drivers, such as vfio-pci make use of only a few interfaces
437 	- The init/release callbacks are issued when vfio_device is initialized
440 	- The open/close device callbacks are issued when the first
444 	- The ioctl callback provides a direct pass through for some VFIO_DEVICE_*
447 	- The [un]bind_iommufd callbacks are issued when the device is bound to
450 	- The [de]attach_ioas callback is issued when the device is attached to
455 	- The read/write/mmap callbacks implement the device region access defined
458 	- The request callback is issued when device is going to be unregistered,
461 	- The dma_unmap callback is issued when a range of iovas are unmapped
468 -------------------------------
472 1) On older systems (POWER7 with P5IOC2/IODA1) only one IOMMU group per
473    container is supported as an IOMMU table is allocated at the boot time,
474    one table per a IOMMU group which is a Partitionable Endpoint (PE)
478    to remove this limitation and have multiple IOMMU groups per a VFIO
481 2) The hardware supports so called DMA windows - the PCI address range
494    error recovery. A PE may be a single or multi-function IOA (IO Adapter), a
495    function of a multi-function IOA, or multiple IOAs (possibly including
524 	/* Enable the IOMMU model we want */
527 	/* Get addition sPAPR IOMMU info */
561 	 * PE, and put child devices belonging to same IOMMU group to the
575 	/* Inject EEH error, which is expected to be caused by 32-bits
629 5) There is v2 of SPAPR TCE IOMMU. It deprecates VFIO_IOMMU_ENABLE/
632    (which are unsupported in v1 IOMMU).
637    The v2 IOMMU splits accounting and pinning into separate operations:
639    - VFIO_IOMMU_SPAPR_REGISTER_MEMORY/VFIO_IOMMU_SPAPR_UNREGISTER_MEMORY ioctls
646    - VFIO_IOMMU_MAP_DMA/VFIO_IOMMU_UNMAP_DMA ioctls only update the actual
647      IOMMU table and do not do pinning; instead these check that the userspace
648      address is from pre-registered range.
658    be as big as entire RAM, use different page size, it is optional - guests
659    create those in run-time if the guest driver supports 64bit DMA.
671 -------------------------------------------------------------------------------
678    possible for multi-function devices to have backdoors between
682    IOMMU driver to group multi-function PCI devices together
683    (iommu=group_mf).  The latter we can't prevent, but the IOMMU should
684    still provide isolation.  For PCI, SR-IOV Virtual Functions are the
688 .. [3] As always there are trade-offs to virtual machine device
690    future IOMMU technologies will reduce some, but maybe not all, of
691    these trade-offs.
694    from either function of the device are indistinguishable to the iommu::
696 	-[0000:00]-+-1e.0-[06]--+-0d.0
697 				\-0d.1
701 .. [5] Nested translation is an IOMMU feature which supports two stage
703    in IOMMU virtualization.
705 .. [6] PASID stands for Process Address Space ID, introduced by PCI