xref: /openbmc/qemu/docs/pcie_sriov.txt (revision 9951ba94)
1PCI SR/IOV EMULATION SUPPORT
2============================
3
4Description
5===========
6SR/IOV (Single Root I/O Virtualization) is an optional extended capability
7of a PCI Express device. It allows a single physical function (PF) to appear as multiple
8virtual functions (VFs) for the main purpose of eliminating software
9overhead in I/O from virtual machines.
10
11QEMU now implements the basic common functionality to enable an emulated device
12to support SR/IOV. Yet no fully implemented devices exists in QEMU, but a
13proof-of-concept hack of the Intel igb can be found here:
14
15git://github.com/knuto/qemu.git sriov_patches_v5
16
17Implementation
18==============
19Implementing emulation of an SR/IOV capable device typically consists of
20implementing support for two types of device classes; the "normal" physical device
21(PF) and the virtual device (VF). From QEMU's perspective, the VFs are just
22like other devices, except that some of their properties are derived from
23the PF.
24
25A virtual function is different from a physical function in that the BAR
26space for all VFs are defined by the BAR registers in the PFs SR/IOV
27capability. All VFs have the same BARs and BAR sizes.
28
29Accesses to these virtual BARs then is computed as
30
31   <VF BAR start> + <VF number> * <BAR sz> + <offset>
32
33From our emulation perspective this means that there is a separate call for
34setting up a BAR for a VF.
35
361) To enable SR/IOV support in the PF, it must be a PCI Express device so
37   you would need to add a PCI Express capability in the normal PCI
38   capability list. You might also want to add an ARI (Alternative
39   Routing-ID Interpretation) capability to indicate that your device
40   supports functions beyond it's "own" function space (0-7),
41   which is necessary to support more than 7 functions, or
42   if functions extends beyond offset 7 because they are placed at an
43   offset > 1 or have stride > 1.
44
45   ...
46   #include "hw/pci/pcie.h"
47   #include "hw/pci/pcie_sriov.h"
48
49   pci_your_pf_dev_realize( ... )
50   {
51      ...
52      int ret = pcie_endpoint_cap_init(d, 0x70);
53      ...
54      pcie_ari_init(d, 0x100, 1);
55      ...
56
57      /* Add and initialize the SR/IOV capability */
58      pcie_sriov_pf_init(d, 0x200, "your_virtual_dev",
59                       vf_devid, initial_vfs, total_vfs,
60                       fun_offset, stride);
61
62      /* Set up individual VF BARs (parameters as for normal BARs) */
63      pcie_sriov_pf_init_vf_bar( ... )
64      ...
65   }
66
67   For cleanup, you simply call:
68
69      pcie_sriov_pf_exit(device);
70
71   which will delete all the virtual functions and associated resources.
72
732) Similarly in the implementation of the virtual function, you need to
74   make it a PCI Express device and add a similar set of capabilities
75   except for the SR/IOV capability. Then you need to set up the VF BARs as
76   subregions of the PFs SR/IOV VF BARs by calling
77   pcie_sriov_vf_register_bar() instead of the normal pci_register_bar() call:
78
79   pci_your_vf_dev_realize( ... )
80   {
81      ...
82      int ret = pcie_endpoint_cap_init(d, 0x60);
83      ...
84      pcie_ari_init(d, 0x100, 1);
85      ...
86      memory_region_init(mr, ... )
87      pcie_sriov_vf_register_bar(d, bar_nr, mr);
88      ...
89   }
90
91Testing on Linux guest
92======================
93The easiest is if your device driver supports sysfs based SR/IOV
94enabling. Support for this was added in kernel v.3.8, so not all drivers
95support it yet.
96
97To enable 4 VFs for a device at 01:00.0:
98
99	modprobe yourdriver
100	echo 4 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs
101
102You should now see 4 VFs with lspci.
103To turn SR/IOV off again - the standard requires you to turn it off before you can enable
104another VF count, and the emulation enforces this:
105
106	echo 0 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs
107
108Older drivers typically provide a max_vfs module parameter
109to enable it at load time:
110
111	modprobe yourdriver max_vfs=4
112
113To disable the VFs again then, you simply have to unload the driver:
114
115	rmmod yourdriver
116