1.. SPDX-License-Identifier: GPL-2.0-only
2.. include:: <isonum.txt>
3
4=====================
5VFIO Mediated devices
6=====================
7
8:Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved.
9:Author: Neo Jia <cjia@nvidia.com>
10:Author: Kirti Wankhede <kwankhede@nvidia.com>
11
12
13
14Virtual Function I/O (VFIO) Mediated devices[1]
15===============================================
16
17The number of use cases for virtualizing DMA devices that do not have built-in
18SR_IOV capability is increasing. Previously, to virtualize such devices,
19developers had to create their own management interfaces and APIs, and then
20integrate them with user space software. To simplify integration with user space
21software, we have identified common requirements and a unified management
22interface for such devices.
23
24The VFIO driver framework provides unified APIs for direct device access. It is
25an IOMMU/device-agnostic framework for exposing direct device access to user
26space in a secure, IOMMU-protected environment. This framework is used for
27multiple devices, such as GPUs, network adapters, and compute accelerators. With
28direct device access, virtual machines or user space applications have direct
29access to the physical device. This framework is reused for mediated devices.
30
31The mediated core driver provides a common interface for mediated device
32management that can be used by drivers of different devices. This module
33provides a generic interface to perform these operations:
34
35* Create and destroy a mediated device
36* Add a mediated device to and remove it from a mediated bus driver
37* Add a mediated device to and remove it from an IOMMU group
38
39The mediated core driver also provides an interface to register a bus driver.
40For example, the mediated VFIO mdev driver is designed for mediated devices and
41supports VFIO APIs. The mediated bus driver adds a mediated device to and
42removes it from a VFIO group.
43
44The following high-level block diagram shows the main components and interfaces
45in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM
46devices as examples, as these devices are the first devices to use this module::
47
48     +---------------+
49     |               |
50     | +-----------+ |  mdev_register_driver() +--------------+
51     | |           | +<------------------------+              |
52     | |  mdev     | |                         |              |
53     | |  bus      | +------------------------>+ vfio_mdev.ko |<-> VFIO user
54     | |  driver   | |     probe()/remove()    |              |    APIs
55     | |           | |                         +--------------+
56     | +-----------+ |
57     |               |
58     |  MDEV CORE    |
59     |   MODULE      |
60     |   mdev.ko     |
61     | +-----------+ |  mdev_register_device() +--------------+
62     | |           | +<------------------------+              |
63     | |           | |                         |  nvidia.ko   |<-> physical
64     | |           | +------------------------>+              |    device
65     | |           | |        callbacks        +--------------+
66     | | Physical  | |
67     | |  device   | |  mdev_register_device() +--------------+
68     | | interface | |<------------------------+              |
69     | |           | |                         |  i915.ko     |<-> physical
70     | |           | +------------------------>+              |    device
71     | |           | |        callbacks        +--------------+
72     | |           | |
73     | |           | |  mdev_register_device() +--------------+
74     | |           | +<------------------------+              |
75     | |           | |                         | ccw_device.ko|<-> physical
76     | |           | +------------------------>+              |    device
77     | |           | |        callbacks        +--------------+
78     | +-----------+ |
79     +---------------+
80
81
82Registration Interfaces
83=======================
84
85The mediated core driver provides the following types of registration
86interfaces:
87
88* Registration interface for a mediated bus driver
89* Physical device driver interface
90
91Registration Interface for a Mediated Bus Driver
92------------------------------------------------
93
94The registration interface for a mediated device driver provides the following
95structure to represent a mediated device's driver::
96
97     /*
98      * struct mdev_driver [2] - Mediated device's driver
99      * @probe: called when new device created
100      * @remove: called when device removed
101      * @driver: device driver structure
102      */
103     struct mdev_driver {
104	     int  (*probe)  (struct mdev_device *dev);
105	     void (*remove) (struct mdev_device *dev);
106	     struct attribute_group **supported_type_groups;
107	     struct device_driver    driver;
108     };
109
110A mediated bus driver for mdev should use this structure in the function calls
111to register and unregister itself with the core driver:
112
113* Register::
114
115    int mdev_register_driver(struct mdev_driver *drv);
116
117* Unregister::
118
119    void mdev_unregister_driver(struct mdev_driver *drv);
120
121The mediated bus driver's probe function should create a vfio_device on top of
122the mdev_device and connect it to an appropriate implementation of
123vfio_device_ops.
124
125When a driver wants to add the GUID creation sysfs to an existing device it has
126probe'd to then it should call::
127
128    int mdev_register_device(struct device *dev,
129                             struct mdev_driver *mdev_driver);
130
131This will provide the 'mdev_supported_types/XX/create' files which can then be
132used to trigger the creation of a mdev_device. The created mdev_device will be
133attached to the specified driver.
134
135When the driver needs to remove itself it calls::
136
137    void mdev_unregister_device(struct device *dev);
138
139Which will unbind and destroy all the created mdevs and remove the sysfs files.
140
141Mediated Device Management Interface Through sysfs
142==================================================
143
144The management interface through sysfs enables user space software, such as
145libvirt, to query and configure mediated devices in a hardware-agnostic fashion.
146This management interface provides flexibility to the underlying physical
147device's driver to support features such as:
148
149* Mediated device hot plug
150* Multiple mediated devices in a single virtual machine
151* Multiple mediated devices from different physical devices
152
153Links in the mdev_bus Class Directory
154-------------------------------------
155The /sys/class/mdev_bus/ directory contains links to devices that are registered
156with the mdev core driver.
157
158Directories and files under the sysfs for Each Physical Device
159--------------------------------------------------------------
160
161::
162
163  |- [parent physical device]
164  |--- Vendor-specific-attributes [optional]
165  |--- [mdev_supported_types]
166  |     |--- [<type-id>]
167  |     |   |--- create
168  |     |   |--- name
169  |     |   |--- available_instances
170  |     |   |--- device_api
171  |     |   |--- description
172  |     |   |--- [devices]
173  |     |--- [<type-id>]
174  |     |   |--- create
175  |     |   |--- name
176  |     |   |--- available_instances
177  |     |   |--- device_api
178  |     |   |--- description
179  |     |   |--- [devices]
180  |     |--- [<type-id>]
181  |          |--- create
182  |          |--- name
183  |          |--- available_instances
184  |          |--- device_api
185  |          |--- description
186  |          |--- [devices]
187
188* [mdev_supported_types]
189
190  The list of currently supported mediated device types and their details.
191
192  [<type-id>], device_api, and available_instances are mandatory attributes
193  that should be provided by vendor driver.
194
195* [<type-id>]
196
197  The [<type-id>] name is created by adding the device driver string as a prefix
198  to the string provided by the vendor driver. This format of this name is as
199  follows::
200
201	sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);
202
203  (or using mdev_parent_dev(mdev) to arrive at the parent device outside
204  of the core mdev code)
205
206* device_api
207
208  This attribute should show which device API is being created, for example,
209  "vfio-pci" for a PCI device.
210
211* available_instances
212
213  This attribute should show the number of devices of type <type-id> that can be
214  created.
215
216* [device]
217
218  This directory contains links to the devices of type <type-id> that have been
219  created.
220
221* name
222
223  This attribute should show human readable name. This is optional attribute.
224
225* description
226
227  This attribute should show brief features/description of the type. This is
228  optional attribute.
229
230Directories and Files Under the sysfs for Each mdev Device
231----------------------------------------------------------
232
233::
234
235  |- [parent phy device]
236  |--- [$MDEV_UUID]
237         |--- remove
238         |--- mdev_type {link to its type}
239         |--- vendor-specific-attributes [optional]
240
241* remove (write only)
242
243Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can
244fail the remove() callback if that device is active and the vendor driver
245doesn't support hot unplug.
246
247Example::
248
249	# echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove
250
251Mediated device Hot plug
252------------------------
253
254Mediated devices can be created and assigned at runtime. The procedure to hot
255plug a mediated device is the same as the procedure to hot plug a PCI device.
256
257Translation APIs for Mediated Devices
258=====================================
259
260The following APIs are provided for translating user pfn to host pfn in a VFIO
261driver::
262
263	int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova,
264				  int npage, int prot, struct page **pages);
265
266	void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova,
267				    int npage);
268
269These functions call back into the back-end IOMMU module by using the pin_pages
270and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently
271these callbacks are supported in the TYPE1 IOMMU module. To enable them for
272other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide
273these two callback functions.
274
275Using the Sample Code
276=====================
277
278mtty.c in samples/vfio-mdev/ directory is a sample driver program to
279demonstrate how to use the mediated device framework.
280
281The sample driver creates an mdev device that simulates a serial port over a PCI
282card.
283
2841. Build and load the mtty.ko module.
285
286   This step creates a dummy device, /sys/devices/virtual/mtty/mtty/
287
288   Files in this device directory in sysfs are similar to the following::
289
290     # tree /sys/devices/virtual/mtty/mtty/
291        /sys/devices/virtual/mtty/mtty/
292        |-- mdev_supported_types
293        |   |-- mtty-1
294        |   |   |-- available_instances
295        |   |   |-- create
296        |   |   |-- device_api
297        |   |   |-- devices
298        |   |   `-- name
299        |   `-- mtty-2
300        |       |-- available_instances
301        |       |-- create
302        |       |-- device_api
303        |       |-- devices
304        |       `-- name
305        |-- mtty_dev
306        |   `-- sample_mtty_dev
307        |-- power
308        |   |-- autosuspend_delay_ms
309        |   |-- control
310        |   |-- runtime_active_time
311        |   |-- runtime_status
312        |   `-- runtime_suspended_time
313        |-- subsystem -> ../../../../class/mtty
314        `-- uevent
315
3162. Create a mediated device by using the dummy device that you created in the
317   previous step::
318
319     # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" >	\
320              /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create
321
3223. Add parameters to qemu-kvm::
323
324     -device vfio-pci,\
325      sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001
326
3274. Boot the VM.
328
329   In the Linux guest VM, with no hardware on the host, the device appears
330   as  follows::
331
332     # lspci -s 00:05.0 -xxvv
333     00:05.0 Serial controller: Device 4348:3253 (rev 10) (prog-if 02 [16550])
334             Subsystem: Device 4348:3253
335             Physical Slot: 5
336             Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
337     Stepping- SERR- FastB2B- DisINTx-
338             Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
339     <TAbort- <MAbort- >SERR- <PERR- INTx-
340             Interrupt: pin A routed to IRQ 10
341             Region 0: I/O ports at c150 [size=8]
342             Region 1: I/O ports at c158 [size=8]
343             Kernel driver in use: serial
344     00: 48 43 53 32 01 00 00 02 10 02 00 07 00 00 00 00
345     10: 51 c1 00 00 59 c1 00 00 00 00 00 00 00 00 00 00
346     20: 00 00 00 00 00 00 00 00 00 00 00 00 48 43 53 32
347     30: 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00
348
349     In the Linux guest VM, dmesg output for the device is as follows:
350
351     serial 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
352     0000:00:05.0: ttyS1 at I/O 0xc150 (irq = 10) is a 16550A
353     0000:00:05.0: ttyS2 at I/O 0xc158 (irq = 10) is a 16550A
354
355
3565. In the Linux guest VM, check the serial ports::
357
358     # setserial -g /dev/ttyS*
359     /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
360     /dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10
361     /dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10
362
3636. Using minicom or any terminal emulation program, open port /dev/ttyS1 or
364   /dev/ttyS2 with hardware flow control disabled.
365
3667. Type data on the minicom terminal or send data to the terminal emulation
367   program and read the data.
368
369   Data is loop backed from hosts mtty driver.
370
3718. Destroy the mediated device that you created::
372
373     # echo 1 > /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove
374
375References
376==========
377
3781. See Documentation/driver-api/vfio.rst for more information on VFIO.
3792. struct mdev_driver in include/linux/mdev.h
3803. struct mdev_parent_ops in include/linux/mdev.h
3814. struct vfio_iommu_driver_ops in include/linux/vfio.h
382