1=================================== 2pSeries family boards (``pseries``) 3=================================== 4 5The Power machine para-virtualized environment described by the Linux on Power 6Architecture Reference ([LoPAR]_) document is called pSeries. This environment 7is also known as sPAPR, System p guests, or simply Power Linux guests (although 8it is capable of running other operating systems, such as AIX). 9 10Even though pSeries is designed to behave as a guest environment, it is also 11capable of acting as a hypervisor OS, providing, on that role, nested 12virtualization capabilities. 13 14Supported devices 15================= 16 17 * Multi processor support for many Power processors generations: POWER7, 18 POWER7+, POWER8, POWER8NVL, POWER9, and Power10. Support for POWER5+ exists, 19 but its state is unknown. 20 * Interrupt Controller, XICS (POWER8) and XIVE (POWER9 and Power10) 21 * vPHB PCIe Host bridge. 22 * vscsi and vnet devices, compatible with the same devices available on a 23 PowerVM hypervisor with VIOS managing LPARs. 24 * Virtio based devices. 25 * PCIe device pass through. 26 27Missing devices 28=============== 29 30 * SPICE support. 31 32Firmware 33======== 34 35`SLOF <https://github.com/aik/SLOF>`_ (Slimline Open Firmware) is an 36implementation of the `IEEE 1275-1994, Standard for Boot (Initialization 37Configuration) Firmware: Core Requirements and Practices 38<https://standards.ieee.org/standard/1275-1994.html>`_. 39 40QEMU includes a prebuilt image of SLOF which is updated when a more recent 41version is required. 42 43Build directions 44================ 45 46.. code-block:: bash 47 48 ./configure --target-list=ppc64-softmmu && make 49 50Running instructions 51==================== 52 53Someone can select the pSeries machine type by running QEMU with the following 54options: 55 56.. code-block:: bash 57 58 qemu-system-ppc64 -M pseries <other QEMU arguments> 59 60sPAPR devices 61============= 62 63The sPAPR specification defines a set of para-virtualized devices, which are 64also supported by the pSeries machine in QEMU and can be instantiated with the 65``-device`` option: 66 67* ``spapr-vlan`` : a virtual network interface. 68* ``spapr-vscsi`` : a virtual SCSI disk interface. 69* ``spapr-rng`` : a pseudo-device for passing random number generator data to the 70 guest (see the `H_RANDOM hypercall feature 71 <https://wiki.qemu.org/Features/HRandomHypercall>`_ for details). 72* ``spapr-vty``: a virtual teletype. 73* ``spapr-pci-host-bridge``: a PCI host bridge. 74* ``tpm-spapr``: a Trusted Platform Module (TPM). 75* ``spapr-tpm-proxy``: a TPM proxy. 76 77These are compatible with the devices historically available for use when 78running the IBM PowerVM hypervisor with LPARs. 79 80However, since these devices have originally been specified with another 81hypervisor and non-Linux guests in mind, you should use the virtio counterparts 82(virtio-net, virtio-blk/scsi and virtio-rng for instance) if possible instead, 83since they will most probably give you better performance with Linux guests in a 84QEMU environment. 85 86The pSeries machine in QEMU is always instantiated with the following devices: 87 88* A NVRAM device (``spapr-nvram``). 89* A virtual teletype (``spapr-vty``). 90* A PCI host bridge (``spapr-pci-host-bridge``). 91 92Hence, it is not needed to add them manually, unless you use the ``-nodefaults`` 93command line option in QEMU. 94 95In the case of the default ``spapr-nvram`` device, if someone wants to make the 96contents of the NVRAM device persistent, they will need to specify a PFLASH 97device when starting QEMU, i.e. either use 98``-drive if=pflash,file=<filename>,format=raw`` to set the default PFLASH 99device, or specify one with an ID 100(``-drive if=none,file=<filename>,format=raw,id=pfid``) and pass that ID to the 101NVRAM device with ``-global spapr-nvram.drive=pfid``. 102 103sPAPR specification 104------------------- 105 106The main source of documentation on the sPAPR standard is the [LoPAR]_ document. 107However, documentation specific to QEMU's implementation of the specification 108can also be found in QEMU documentation: 109 110.. toctree:: 111 :maxdepth: 1 112 113 ../../specs/ppc-spapr-hcalls.rst 114 ../../specs/ppc-spapr-numa.rst 115 ../../specs/ppc-spapr-xive.rst 116 117Other documentation available in QEMU docs directory: 118 119* Hot plug (``/docs/specs/ppc-spapr-hotplug.txt``). 120* Hypervisor calls needed by the Ultravisor 121 (``/docs/specs/ppc-spapr-uv-hcalls.txt``). 122 123Switching between the KVM-PR and KVM-HV kernel module 124===================================================== 125 126Currently, there are two implementations of KVM on Power, ``kvm_hv.ko`` and 127``kvm_pr.ko``. 128 129 130If a host supports both KVM modes, and both KVM kernel modules are loaded, it is 131possible to switch between the two modes with the ``kvm-type`` parameter: 132 133* Use ``qemu-system-ppc64 -M pseries,accel=kvm,kvm-type=PR`` to use the 134 ``kvm_pr.ko`` kernel module. 135* Use ``qemu-system-ppc64 -M pseries,accel=kvm,kvm-type=HV`` to use ``kvm_hv.ko`` 136 instead. 137 138KVM-PR 139------ 140 141KVM-PR uses the so-called **PR**\ oblem state of the PPC CPUs to run the guests, 142i.e. the virtual machine is run in user mode and all privileged instructions 143trap and have to be emulated by the host. That means you can run KVM-PR inside 144a pSeries guest (or a PowerVM LPAR for that matter), and that is where it has 145originated, as historically (prior to POWER7) it was not possible to run Linux 146on hypervisor mode on a Power processor (this function was restricted to 147PowerVM, the IBM proprietary hypervisor). 148 149Because all privileged instructions are trapped, guests that use a lot of 150privileged instructions run quite slow with KVM-PR. On the other hand, because 151of that, this kernel module can run on pretty much every PPC hardware, and is 152able to emulate a lot of guests CPUs. This module can even be used to run other 153PowerPC guests like an emulated PowerMac. 154 155As KVM-PR can be run inside a pSeries guest, it can also provide nested 156virtualization capabilities (i.e. running a guest from within a guest). 157 158It is important to notice that, as KVM-HV provides a much better execution 159performance, maintenance work has been much more focused on it in the past 160years. Maintenance for KVM-PR has been minimal. 161 162In order to run KVM-PR guests with POWER9 processors, someone will need to start 163QEMU with ``kernel_irqchip=off`` command line option. 164 165KVM-HV 166------ 167 168KVM-HV uses the hypervisor mode of more recent Power processors, that allow 169access to the bare metal hardware directly. Although POWER7 had this capability, 170it was only starting with POWER8 that this was officially supported by IBM. 171 172Originally, KVM-HV was only available when running on a PowerNV platform (a.k.a. 173Power bare metal). Although it runs on a PowerNV platform, it can only be used 174to start pSeries guests. As the pSeries guest doesn't have access to the 175hypervisor mode of the Power CPU, it wasn't possible to run KVM-HV on a guest. 176This limitation has been lifted, and now it is possible to run KVM-HV inside 177pSeries guests as well, making nested virtualization possible with KVM-HV. 178 179As KVM-HV has access to privileged instructions, guests that use a lot of these 180can run much faster than with KVM-PR. On the other hand, the guest CPU has to be 181of the same type as the host CPU this way, e.g. it is not possible to specify an 182embedded PPC CPU for the guest with KVM-HV. However, there is at least the 183possibility to run the guest in a backward-compatibility mode of the previous 184CPUs generations, e.g. you can run a POWER7 guest on a POWER8 host by using 185``-cpu POWER8,compat=power7`` as parameter to QEMU. 186 187Modules support 188=============== 189 190As noticed in the sections above, each module can run in a different 191environment. The following table shows with which environment each module can 192run. As long as you are in a supported environment, you can run KVM-PR or KVM-HV 193nested. Combinations not shown in the table are not available. 194 195+--------------+------------+------+-------------------+----------+--------+ 196| Platform | Host type | Bits | Page table format | KVM-HV | KVM-PR | 197+==============+============+======+===================+==========+========+ 198| PowerNV | bare metal | 32 | hash | no | yes | 199| | | +-------------------+----------+--------+ 200| | | | radix | N/A | N/A | 201| | +------+-------------------+----------+--------+ 202| | | 64 | hash | yes | yes | 203| | | +-------------------+----------+--------+ 204| | | | radix | yes | no | 205+--------------+------------+------+-------------------+----------+--------+ 206| pSeries [1]_ | PowerNV | 32 | hash | no | yes | 207| | | +-------------------+----------+--------+ 208| | | | radix | N/A | N/A | 209| | +------+-------------------+----------+--------+ 210| | | 64 | hash | no | yes | 211| | | +-------------------+----------+--------+ 212| | | | radix | yes [2]_ | no | 213| +------------+------+-------------------+----------+--------+ 214| | PowerVM | 32 | hash | no | yes | 215| | | +-------------------+----------+--------+ 216| | | | radix | N/A | N/A | 217| | +------+-------------------+----------+--------+ 218| | | 64 | hash | no | yes | 219| | | +-------------------+----------+--------+ 220| | | | radix [3]_ | no | yes | 221+--------------+------------+------+-------------------+----------+--------+ 222 223.. [1] On POWER9 DD2.1 processors, the page table format on the host and guest 224 must be the same. 225 226.. [2] KVM-HV cannot run nested on POWER8 machines. 227 228.. [3] Introduced on Power10 machines. 229 230 231POWER (PAPR) Protected Execution Facility (PEF) 232----------------------------------------------- 233 234Protected Execution Facility (PEF), also known as Secure Guest support 235is a feature found on IBM POWER9 and POWER10 processors. 236 237If a suitable firmware including an Ultravisor is installed, it adds 238an extra memory protection mode to the CPU. The ultravisor manages a 239pool of secure memory which cannot be accessed by the hypervisor. 240 241When this feature is enabled in QEMU, a guest can use ultracalls to 242enter "secure mode". This transfers most of its memory to secure 243memory, where it cannot be eavesdropped by a compromised hypervisor. 244 245Launching 246^^^^^^^^^ 247 248To launch a guest which will be permitted to enter PEF secure mode:: 249 250 $ qemu-system-ppc64 \ 251 -object pef-guest,id=pef0 \ 252 -machine confidential-guest-support=pef0 \ 253 ... 254 255Live Migration 256^^^^^^^^^^^^^^ 257 258Live migration is not yet implemented for PEF guests. For 259consistency, QEMU currently prevents migration if the PEF feature is 260enabled, whether or not the guest has actually entered secure mode. 261 262 263Maintainer contact information 264============================== 265 266Cédric Le Goater <clg@kaod.org> 267 268Daniel Henrique Barboza <danielhb413@gmail.com> 269 270.. [LoPAR] `Linux on Power Architecture Reference document (LoPAR) revision 271 2.9 <https://openpowerfoundation.org/wp-content/uploads/2020/07/LoPAR-20200812.pdf>`_. 272