1XIVE for sPAPR (pseries machines) 2================================= 3 4The POWER9 processor comes with a new interrupt controller 5architecture, called XIVE as "eXternal Interrupt Virtualization 6Engine". It supports a larger number of interrupt sources and offers 7virtualization features which enables the HW to deliver interrupts 8directly to virtual processors without hypervisor assistance. 9 10A QEMU ``pseries`` machine (which is PAPR compliant) using POWER9 11processors can run under two interrupt modes: 12 13- *Legacy Compatibility Mode* 14 15 the hypervisor provides identical interfaces and similar 16 functionality to PAPR+ Version 2.7. This is the default mode 17 18 It is also referred as *XICS* in QEMU. 19 20- *XIVE native exploitation mode* 21 22 the hypervisor provides new interfaces to manage the XIVE control 23 structures, and provides direct control for interrupt management 24 through MMIO pages. 25 26Which interrupt modes can be used by the machine is negotiated with 27the guest O/S during the Client Architecture Support negotiation 28sequence. The two modes are mutually exclusive. 29 30Both interrupt mode share the same IRQ number space. See below for the 31layout. 32 33CAS Negotiation 34--------------- 35 36QEMU advertises the supported interrupt modes in the device tree 37property ``ibm,arch-vec-5-platform-support`` in byte 23 and the OS 38Selection for XIVE is indicated in the ``ibm,architecture-vec-5`` 39property byte 23. 40 41The interrupt modes supported by the machine depend on the CPU type 42(POWER9 is required for XIVE) but also on the machine property 43``ic-mode`` which can be set on the command line. It can take the 44following values: ``xics``, ``xive``, and ``dual`` which is the 45default mode. ``dual`` means that both modes XICS **and** XIVE are 46supported and if the guest OS supports XIVE, this mode will be 47selected. 48 49The choosen interrupt mode is activated after a reconfiguration done 50in a machine reset. 51 52KVM negotiation 53--------------- 54 55When the guest starts under KVM, the capabilities of the host kernel 56and QEMU are also negotiated. Depending on the version of the host 57kernel, KVM will advertise the XIVE capability to QEMU or not. 58 59Nevertheless, the available interrupt modes in the machine should not 60depend on the XIVE KVM capability of the host. On older kernels 61without XIVE KVM support, QEMU will use the emulated XIVE device as a 62fallback and on newer kernels (>=5.2), the KVM XIVE device. 63 64XIVE native exploitation mode is not supported for KVM nested guests, 65VMs running under a L1 hypervisor (KVM on pSeries). In that case, the 66hypervisor will not advertise the KVM capability and QEMU will use the 67emulated XIVE device, same as for older versions of KVM. 68 69As a final refinement, the user can also switch the use of the KVM 70device with the machine option ``kernel_irqchip``. 71 72 73XIVE support in KVM 74~~~~~~~~~~~~~~~~~~~ 75 76For guest OSes supporting XIVE, the resulting interrupt modes on host 77kernels with XIVE KVM support are the following: 78 79============== ============= ============= ================ 80ic-mode kernel_irqchip 81-------------- ---------------------------------------------- 82/ allowed off on 83 (default) 84============== ============= ============= ================ 85dual (default) XIVE KVM XIVE emul. XIVE KVM 86xive XIVE KVM XIVE emul. XIVE KVM 87xics XICS KVM XICS emul. XICS KVM 88============== ============= ============= ================ 89 90For legacy guest OSes without XIVE support, the resulting interrupt 91modes are the following: 92 93============== ============= ============= ================ 94ic-mode kernel_irqchip 95-------------- ---------------------------------------------- 96/ allowed off on 97 (default) 98============== ============= ============= ================ 99dual (default) XICS KVM XICS emul. XICS KVM 100xive QEMU error(3) QEMU error(3) QEMU error(3) 101xics XICS KVM XICS emul. XICS KVM 102============== ============= ============= ================ 103 104(3) QEMU fails at CAS with ``Guest requested unavailable interrupt 105 mode (XICS), either don't set the ic-mode machine property or try 106 ic-mode=xics or ic-mode=dual`` 107 108 109No XIVE support in KVM 110~~~~~~~~~~~~~~~~~~~~~~ 111 112For guest OSes supporting XIVE, the resulting interrupt modes on host 113kernels without XIVE KVM support are the following: 114 115============== ============= ============= ================ 116ic-mode kernel_irqchip 117-------------- ---------------------------------------------- 118/ allowed off on 119 (default) 120============== ============= ============= ================ 121dual (default) XIVE emul.(1) XIVE emul. QEMU error (2) 122xive XIVE emul.(1) XIVE emul. QEMU error (2) 123xics XICS KVM XICS emul. XICS KVM 124============== ============= ============= ================ 125 126 127(1) QEMU warns with ``warning: kernel_irqchip requested but unavailable: 128 IRQ_XIVE capability must be present for KVM`` 129(2) QEMU fails with ``kernel_irqchip requested but unavailable: 130 IRQ_XIVE capability must be present for KVM`` 131 132 133For legacy guest OSes without XIVE support, the resulting interrupt 134modes are the following: 135 136============== ============= ============= ================ 137ic-mode kernel_irqchip 138-------------- ---------------------------------------------- 139/ allowed off on 140 (default) 141============== ============= ============= ================ 142dual (default) QEMU error(4) XICS emul. QEMU error(4) 143xive QEMU error(3) QEMU error(3) QEMU error(3) 144xics XICS KVM XICS emul. XICS KVM 145============== ============= ============= ================ 146 147(3) QEMU fails at CAS with ``Guest requested unavailable interrupt 148 mode (XICS), either don't set the ic-mode machine property or try 149 ic-mode=xics or ic-mode=dual`` 150(4) QEMU/KVM incompatibility due to device destruction in reset. QEMU fails 151 with ``KVM is too old to support ic-mode=dual,kernel-irqchip=on`` 152 153 154XIVE Device tree properties 155--------------------------- 156 157The properties for the PAPR interrupt controller node when the *XIVE 158native exploitation mode* is selected shoud contain: 159 160- ``device_type`` 161 162 value should be "power-ivpe". 163 164- ``compatible`` 165 166 value should be "ibm,power-ivpe". 167 168- ``reg`` 169 170 contains the base address and size of the thread interrupt 171 managnement areas (TIMA), for the User level and for the Guest OS 172 level. Only the Guest OS level is taken into account today. 173 174- ``ibm,xive-eq-sizes`` 175 176 the size of the event queues. One cell per size supported, contains 177 log2 of size, in ascending order. 178 179- ``ibm,xive-lisn-ranges`` 180 181 the IRQ interrupt number ranges assigned to the guest for the IPIs. 182 183The root node also exports : 184 185- ``ibm,plat-res-int-priorities`` 186 187 contains a list of priorities that the hypervisor has reserved for 188 its own use. 189 190IRQ number space 191---------------- 192 193IRQ Number space of the ``pseries`` machine is 8K wide and is the same 194for both interrupt mode. The different ranges are defined as follow : 195 196- ``0x0000 .. 0x0FFF`` 4K CPU IPIs (only used under XIVE) 197- ``0x1000 .. 0x1000`` 1 EPOW 198- ``0x1001 .. 0x1001`` 1 HOTPLUG 199- ``0x1002 .. 0x10FF`` unused 200- ``0x1100 .. 0x11FF`` 256 VIO devices 201- ``0x1200 .. 0x127F`` 32x4 LSIs for PHB devices 202- ``0x1280 .. 0x12FF`` unused 203- ``0x1300 .. 0x1FFF`` PHB MSIs (dynamically allocated) 204 205Monitoring XIVE 206--------------- 207 208The state of the XIVE interrupt controller can be queried through the 209monitor commands ``info pic``. The output comes in two parts. 210 211First, the state of the thread interrupt context registers is dumped 212for each CPU : 213 214:: 215 216 (qemu) info pic 217 CPU[0000]: QW NSR CPPR IPB LSMFB ACK# INC AGE PIPR W2 218 CPU[0000]: USER 00 00 00 00 00 00 00 00 00000000 219 CPU[0000]: OS 00 ff 00 00 ff 00 ff ff 80000400 220 CPU[0000]: POOL 00 00 00 00 00 00 00 00 00000000 221 CPU[0000]: PHYS 00 00 00 00 00 00 00 ff 00000000 222 ... 223 224In the case of a ``pseries`` machine, QEMU acts as the hypervisor and only 225the O/S and USER register rings make sense. ``W2`` contains the vCPU CAM 226line which is set to the VP identifier. 227 228Then comes the routing information which aggregates the EAS and the 229END configuration: 230 231:: 232 233 ... 234 LISN PQ EISN CPU/PRIO EQ 235 00000000 MSI -- 00000010 0/6 380/16384 @1fe3e0000 ^1 [ 80000010 ... ] 236 00000001 MSI -- 00000010 1/6 305/16384 @1fc230000 ^1 [ 80000010 ... ] 237 00000002 MSI -- 00000010 2/6 220/16384 @1fc2f0000 ^1 [ 80000010 ... ] 238 00000003 MSI -- 00000010 3/6 201/16384 @1fc390000 ^1 [ 80000010 ... ] 239 00000004 MSI -Q M 00000000 240 00000005 MSI -Q M 00000000 241 00000006 MSI -Q M 00000000 242 00000007 MSI -Q M 00000000 243 00001000 MSI -- 00000012 0/6 380/16384 @1fe3e0000 ^1 [ 80000010 ... ] 244 00001001 MSI -- 00000013 0/6 380/16384 @1fe3e0000 ^1 [ 80000010 ... ] 245 00001100 MSI -- 00000100 1/6 305/16384 @1fc230000 ^1 [ 80000010 ... ] 246 00001101 MSI -Q M 00000000 247 00001200 LSI -Q M 00000000 248 00001201 LSI -Q M 00000000 249 00001202 LSI -Q M 00000000 250 00001203 LSI -Q M 00000000 251 00001300 MSI -- 00000102 1/6 305/16384 @1fc230000 ^1 [ 80000010 ... ] 252 00001301 MSI -- 00000103 2/6 220/16384 @1fc2f0000 ^1 [ 80000010 ... ] 253 00001302 MSI -- 00000104 3/6 201/16384 @1fc390000 ^1 [ 80000010 ... ] 254 255The source information and configuration: 256 257- The ``LISN`` column outputs the interrupt number of the source in 258 range ``[ 0x0 ... 0x1FFF ]`` and its type : ``MSI`` or ``LSI`` 259- The ``PQ`` column reflects the state of the PQ bits of the source : 260 261 - ``--`` source is ready to take events 262 - ``P-`` an event was sent and an EOI is PENDING 263 - ``PQ`` an event was QUEUED 264 - ``-Q`` source is OFF 265 266 a ``M`` indicates that source is *MASKED* at the EAS level, 267 268The targeting configuration : 269 270- The ``EISN`` column is the event data that will be queued in the event 271 queue of the O/S. 272- The ``CPU/PRIO`` column is the tuple defining the CPU number and 273 priority queue serving the source. 274- The ``EQ`` column outputs : 275 276 - the current index of the event queue/ the max number of entries 277 - the O/S event queue address 278 - the toggle bit 279 - the last entries that were pushed in the event queue. 280