1Arm CPU Features 2================ 3 4CPU features are optional features that a CPU of supporting type may 5choose to implement or not. In QEMU, optional CPU features have 6corresponding boolean CPU proprieties that, when enabled, indicate 7that the feature is implemented, and, conversely, when disabled, 8indicate that it is not implemented. An example of an Arm CPU feature 9is the Performance Monitoring Unit (PMU). CPU types such as the 10Cortex-A15 and the Cortex-A57, which respectively implement Arm 11architecture reference manuals ARMv7-A and ARMv8-A, may both optionally 12implement PMUs. For example, if a user wants to use a Cortex-A15 without 13a PMU, then the ``-cpu`` parameter should contain ``pmu=off`` on the QEMU 14command line, i.e. ``-cpu cortex-a15,pmu=off``. 15 16As not all CPU types support all optional CPU features, then whether or 17not a CPU property exists depends on the CPU type. For example, CPUs 18that implement the ARMv8-A architecture reference manual may optionally 19support the AArch32 CPU feature, which may be enabled by disabling the 20``aarch64`` CPU property. A CPU type such as the Cortex-A15, which does 21not implement ARMv8-A, will not have the ``aarch64`` CPU property. 22 23QEMU's support may be limited for some CPU features, only partially 24supporting the feature or only supporting the feature under certain 25configurations. For example, the ``aarch64`` CPU feature, which, when 26disabled, enables the optional AArch32 CPU feature, is only supported 27when using the KVM accelerator and when running on a host CPU type that 28supports the feature. While ``aarch64`` currently only works with KVM, 29it could work with TCG. CPU features that are specific to KVM are 30prefixed with "kvm-" and are described in "KVM VCPU Features". 31 32CPU Feature Probing 33=================== 34 35Determining which CPU features are available and functional for a given 36CPU type is possible with the ``query-cpu-model-expansion`` QMP command. 37Below are some examples where ``scripts/qmp/qmp-shell`` (see the top comment 38block in the script for usage) is used to issue the QMP commands. 39 401. Determine which CPU features are available for the ``max`` CPU type 41 (Note, we started QEMU with qemu-system-aarch64, so ``max`` is 42 implementing the ARMv8-A reference manual in this case):: 43 44 (QEMU) query-cpu-model-expansion type=full model={"name":"max"} 45 { "return": { 46 "model": { "name": "max", "props": { 47 "sve1664": true, "pmu": true, "sve1792": true, "sve1920": true, 48 "sve128": true, "aarch64": true, "sve1024": true, "sve": true, 49 "sve640": true, "sve768": true, "sve1408": true, "sve256": true, 50 "sve1152": true, "sve512": true, "sve384": true, "sve1536": true, 51 "sve896": true, "sve1280": true, "sve2048": true 52 }}}} 53 54We see that the ``max`` CPU type has the ``pmu``, ``aarch64``, ``sve``, and many 55``sve<N>`` CPU features. We also see that all the CPU features are 56enabled, as they are all ``true``. (The ``sve<N>`` CPU features are all 57optional SVE vector lengths (see "SVE CPU Properties"). While with TCG 58all SVE vector lengths can be supported, when KVM is in use it's more 59likely that only a few lengths will be supported, if SVE is supported at 60all.) 61 62(2) Let's try to disable the PMU:: 63 64 (QEMU) query-cpu-model-expansion type=full model={"name":"max","props":{"pmu":false}} 65 { "return": { 66 "model": { "name": "max", "props": { 67 "sve1664": true, "pmu": false, "sve1792": true, "sve1920": true, 68 "sve128": true, "aarch64": true, "sve1024": true, "sve": true, 69 "sve640": true, "sve768": true, "sve1408": true, "sve256": true, 70 "sve1152": true, "sve512": true, "sve384": true, "sve1536": true, 71 "sve896": true, "sve1280": true, "sve2048": true 72 }}}} 73 74We see it worked, as ``pmu`` is now ``false``. 75 76(3) Let's try to disable ``aarch64``, which enables the AArch32 CPU feature:: 77 78 (QEMU) query-cpu-model-expansion type=full model={"name":"max","props":{"aarch64":false}} 79 {"error": { 80 "class": "GenericError", "desc": 81 "'aarch64' feature cannot be disabled unless KVM is enabled and 32-bit EL1 is supported" 82 }} 83 84It looks like this feature is limited to a configuration we do not 85currently have. 86 87(4) Let's disable ``sve`` and see what happens to all the optional SVE 88 vector lengths:: 89 90 (QEMU) query-cpu-model-expansion type=full model={"name":"max","props":{"sve":false}} 91 { "return": { 92 "model": { "name": "max", "props": { 93 "sve1664": false, "pmu": true, "sve1792": false, "sve1920": false, 94 "sve128": false, "aarch64": true, "sve1024": false, "sve": false, 95 "sve640": false, "sve768": false, "sve1408": false, "sve256": false, 96 "sve1152": false, "sve512": false, "sve384": false, "sve1536": false, 97 "sve896": false, "sve1280": false, "sve2048": false 98 }}}} 99 100As expected they are now all ``false``. 101 102(5) Let's try probing CPU features for the Cortex-A15 CPU type:: 103 104 (QEMU) query-cpu-model-expansion type=full model={"name":"cortex-a15"} 105 {"return": {"model": {"name": "cortex-a15", "props": {"pmu": true}}}} 106 107Only the ``pmu`` CPU feature is available. 108 109A note about CPU feature dependencies 110------------------------------------- 111 112It's possible for features to have dependencies on other features. I.e. 113it may be possible to change one feature at a time without error, but 114when attempting to change all features at once an error could occur 115depending on the order they are processed. It's also possible changing 116all at once doesn't generate an error, because a feature's dependencies 117are satisfied with other features, but the same feature cannot be changed 118independently without error. For these reasons callers should always 119attempt to make their desired changes all at once in order to ensure the 120collection is valid. 121 122A note about CPU models and KVM 123------------------------------- 124 125Named CPU models generally do not work with KVM. There are a few cases 126that do work, e.g. using the named CPU model ``cortex-a57`` with KVM on a 127seattle host, but mostly if KVM is enabled the ``host`` CPU type must be 128used. This means the guest is provided all the same CPU features as the 129host CPU type has. And, for this reason, the ``host`` CPU type should 130enable all CPU features that the host has by default. Indeed it's even 131a bit strange to allow disabling CPU features that the host has when using 132the ``host`` CPU type, but in the absence of CPU models it's the best we can 133do if we want to launch guests without all the host's CPU features enabled. 134 135Enabling KVM also affects the ``query-cpu-model-expansion`` QMP command. The 136affect is not only limited to specific features, as pointed out in example 137(3) of "CPU Feature Probing", but also to which CPU types may be expanded. 138When KVM is enabled, only the ``max``, ``host``, and current CPU type may be 139expanded. This restriction is necessary as it's not possible to know all 140CPU types that may work with KVM, but it does impose a small risk of users 141experiencing unexpected errors. For example on a seattle, as mentioned 142above, the ``cortex-a57`` CPU type is also valid when KVM is enabled. 143Therefore a user could use the ``host`` CPU type for the current type, but 144then attempt to query ``cortex-a57``, however that query will fail with our 145restrictions. This shouldn't be an issue though as management layers and 146users have been preferring the ``host`` CPU type for use with KVM for quite 147some time. Additionally, if the KVM-enabled QEMU instance running on a 148seattle host is using the ``cortex-a57`` CPU type, then querying ``cortex-a57`` 149will work. 150 151Using CPU Features 152================== 153 154After determining which CPU features are available and supported for a 155given CPU type, then they may be selectively enabled or disabled on the 156QEMU command line with that CPU type:: 157 158 $ qemu-system-aarch64 -M virt -cpu max,pmu=off,sve=on,sve128=on,sve256=on 159 160The example above disables the PMU and enables the first two SVE vector 161lengths for the ``max`` CPU type. Note, the ``sve=on`` isn't actually 162necessary, because, as we observed above with our probe of the ``max`` CPU 163type, ``sve`` is already on by default. Also, based on our probe of 164defaults, it would seem we need to disable many SVE vector lengths, rather 165than only enabling the two we want. This isn't the case, because, as 166disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU 167properties have special semantics (see "SVE CPU Property Parsing 168Semantics"). 169 170KVM VCPU Features 171================= 172 173KVM VCPU features are CPU features that are specific to KVM, such as 174paravirt features or features that enable CPU virtualization extensions. 175The features' CPU properties are only available when KVM is enabled and 176are named with the prefix "kvm-". KVM VCPU features may be probed, 177enabled, and disabled in the same way as other CPU features. Below is 178the list of KVM VCPU features and their descriptions. 179 180``kvm-no-adjvtime`` 181 By default kvm-no-adjvtime is disabled. This means that by default 182 the virtual time adjustment is enabled (vtime is not *not* adjusted). 183 184 When virtual time adjustment is enabled each time the VM transitions 185 back to running state the VCPU's virtual counter is updated to 186 ensure stopped time is not counted. This avoids time jumps 187 surprising guest OSes and applications, as long as they use the 188 virtual counter for timekeeping. However it has the side effect of 189 the virtual and physical counters diverging. All timekeeping based 190 on the virtual counter will appear to lag behind any timekeeping 191 that does not subtract VM stopped time. The guest may resynchronize 192 its virtual counter with other time sources as needed. 193 194 Enable kvm-no-adjvtime to disable virtual time adjustment, also 195 restoring the legacy (pre-5.0) behavior. 196 197``kvm-steal-time`` 198 Since v5.2, kvm-steal-time is enabled by default when KVM is 199 enabled, the feature is supported, and the guest is 64-bit. 200 201 When kvm-steal-time is enabled a 64-bit guest can account for time 202 its CPUs were not running due to the host not scheduling the 203 corresponding VCPU threads. The accounting statistics may influence 204 the guest scheduler behavior and/or be exposed to the guest 205 userspace. 206 207TCG VCPU Features 208================= 209 210TCG VCPU features are CPU features that are specific to TCG. 211Below is the list of TCG VCPU features and their descriptions. 212 213``pauth`` 214 Enable or disable ``FEAT_Pauth`` entirely. 215 216``pauth-impdef`` 217 When ``pauth`` is enabled, select the QEMU implementation defined algorithm. 218 219``pauth-qarma3`` 220 When ``pauth`` is enabled, select the architected QARMA3 algorithm. 221 222Without either ``pauth-impdef`` or ``pauth-qarma3`` enabled, 223the architected QARMA5 algorithm is used. The architected QARMA5 224and QARMA3 algorithms have good cryptographic properties, but can 225be quite slow to emulate. The impdef algorithm used by QEMU is 226non-cryptographic but significantly faster. 227 228SVE CPU Properties 229================== 230 231There are two types of SVE CPU properties: ``sve`` and ``sve<N>``. The first 232is used to enable or disable the entire SVE feature, just as the ``pmu`` 233CPU property completely enables or disables the PMU. The second type 234is used to enable or disable specific vector lengths, where ``N`` is the 235number of bits of the length. The ``sve<N>`` CPU properties have special 236dependencies and constraints, see "SVE CPU Property Dependencies and 237Constraints" below. Additionally, as we want all supported vector lengths 238to be enabled by default, then, in order to avoid overly verbose command 239lines (command lines full of ``sve<N>=off``, for all ``N`` not wanted), we 240provide the parsing semantics listed in "SVE CPU Property Parsing 241Semantics". 242 243SVE CPU Property Dependencies and Constraints 244--------------------------------------------- 245 246 1) At least one vector length must be enabled when ``sve`` is enabled. 247 248 2) If a vector length ``N`` is enabled, then, when KVM is enabled, all 249 smaller, host supported vector lengths must also be enabled. If 250 KVM is not enabled, then only all the smaller, power-of-two vector 251 lengths must be enabled. E.g. with KVM if the host supports all 252 vector lengths up to 512-bits (128, 256, 384, 512), then if ``sve512`` 253 is enabled, the 128-bit vector length, 256-bit vector length, and 254 384-bit vector length must also be enabled. Without KVM, the 384-bit 255 vector length would not be required. 256 257 3) If KVM is enabled then only vector lengths that the host CPU type 258 support may be enabled. If SVE is not supported by the host, then 259 no ``sve*`` properties may be enabled. 260 261SVE CPU Property Parsing Semantics 262---------------------------------- 263 264 1) If SVE is disabled (``sve=off``), then which SVE vector lengths 265 are enabled or disabled is irrelevant to the guest, as the entire 266 SVE feature is disabled and that disables all vector lengths for 267 the guest. However QEMU will still track any ``sve<N>`` CPU 268 properties provided by the user. If later an ``sve=on`` is provided, 269 then the guest will get only the enabled lengths. If no ``sve=on`` 270 is provided and there are explicitly enabled vector lengths, then 271 an error is generated. 272 273 2) If SVE is enabled (``sve=on``), but no ``sve<N>`` CPU properties are 274 provided, then all supported vector lengths are enabled, which when 275 KVM is not in use means including the non-power-of-two lengths, and, 276 when KVM is in use, it means all vector lengths supported by the host 277 processor. 278 279 3) If SVE is enabled, then an error is generated when attempting to 280 disable the last enabled vector length (see constraint (1) of "SVE 281 CPU Property Dependencies and Constraints"). 282 283 4) If one or more vector lengths have been explicitly enabled and at 284 least one of the dependency lengths of the maximum enabled length 285 has been explicitly disabled, then an error is generated (see 286 constraint (2) of "SVE CPU Property Dependencies and Constraints"). 287 288 5) When KVM is enabled, if the host does not support SVE, then an error 289 is generated when attempting to enable any ``sve*`` properties (see 290 constraint (3) of "SVE CPU Property Dependencies and Constraints"). 291 292 6) When KVM is enabled, if the host does support SVE, then an error is 293 generated when attempting to enable any vector lengths not supported 294 by the host (see constraint (3) of "SVE CPU Property Dependencies and 295 Constraints"). 296 297 7) If one or more ``sve<N>`` CPU properties are set ``off``, but no ``sve<N>``, 298 CPU properties are set ``on``, then the specified vector lengths are 299 disabled but the default for any unspecified lengths remains enabled. 300 When KVM is not enabled, disabling a power-of-two vector length also 301 disables all vector lengths larger than the power-of-two length. 302 When KVM is enabled, then disabling any supported vector length also 303 disables all larger vector lengths (see constraint (2) of "SVE CPU 304 Property Dependencies and Constraints"). 305 306 8) If one or more ``sve<N>`` CPU properties are set to ``on``, then they 307 are enabled and all unspecified lengths default to disabled, except 308 for the required lengths per constraint (2) of "SVE CPU Property 309 Dependencies and Constraints", which will even be auto-enabled if 310 they were not explicitly enabled. 311 312 9) If SVE was disabled (``sve=off``), allowing all vector lengths to be 313 explicitly disabled (i.e. avoiding the error specified in (3) of 314 "SVE CPU Property Parsing Semantics"), then if later an ``sve=on`` is 315 provided an error will be generated. To avoid this error, one must 316 enable at least one vector length prior to enabling SVE. 317 318SVE CPU Property Examples 319------------------------- 320 321 1) Disable SVE:: 322 323 $ qemu-system-aarch64 -M virt -cpu max,sve=off 324 325 2) Implicitly enable all vector lengths for the ``max`` CPU type:: 326 327 $ qemu-system-aarch64 -M virt -cpu max 328 329 3) When KVM is enabled, implicitly enable all host CPU supported vector 330 lengths with the ``host`` CPU type:: 331 332 $ qemu-system-aarch64 -M virt,accel=kvm -cpu host 333 334 4) Only enable the 128-bit vector length:: 335 336 $ qemu-system-aarch64 -M virt -cpu max,sve128=on 337 338 5) Disable the 512-bit vector length and all larger vector lengths, 339 since 512 is a power-of-two. This results in all the smaller, 340 uninitialized lengths (128, 256, and 384) defaulting to enabled:: 341 342 $ qemu-system-aarch64 -M virt -cpu max,sve512=off 343 344 6) Enable the 128-bit, 256-bit, and 512-bit vector lengths:: 345 346 $ qemu-system-aarch64 -M virt -cpu max,sve128=on,sve256=on,sve512=on 347 348 7) The same as (6), but since the 128-bit and 256-bit vector 349 lengths are required for the 512-bit vector length to be enabled, 350 then allow them to be auto-enabled:: 351 352 $ qemu-system-aarch64 -M virt -cpu max,sve512=on 353 354 8) Do the same as (7), but by first disabling SVE and then re-enabling it:: 355 356 $ qemu-system-aarch64 -M virt -cpu max,sve=off,sve512=on,sve=on 357 358 9) Force errors regarding the last vector length:: 359 360 $ qemu-system-aarch64 -M virt -cpu max,sve128=off 361 $ qemu-system-aarch64 -M virt -cpu max,sve=off,sve128=off,sve=on 362 363SVE CPU Property Recommendations 364-------------------------------- 365 366The examples in "SVE CPU Property Examples" exhibit many ways to select 367vector lengths which developers may find useful in order to avoid overly 368verbose command lines. However, the recommended way to select vector 369lengths is to explicitly enable each desired length. Therefore only 370example's (1), (4), and (6) exhibit recommended uses of the properties. 371 372SME CPU Property Examples 373------------------------- 374 375 1) Disable SME:: 376 377 $ qemu-system-aarch64 -M virt -cpu max,sme=off 378 379 2) Implicitly enable all vector lengths for the ``max`` CPU type:: 380 381 $ qemu-system-aarch64 -M virt -cpu max 382 383 3) Only enable the 256-bit vector length:: 384 385 $ qemu-system-aarch64 -M virt -cpu max,sme256=on 386 387 3) Enable the 256-bit and 1024-bit vector lengths:: 388 389 $ qemu-system-aarch64 -M virt -cpu max,sme256=on,sme1024=on 390 391 4) Disable the 512-bit vector length. This results in all the other 392 lengths supported by ``max`` defaulting to enabled 393 (128, 256, 1024 and 2048):: 394 395 $ qemu-system-aarch64 -M virt -cpu max,sve512=off 396 397SVE User-mode Default Vector Length Property 398-------------------------------------------- 399 400For qemu-aarch64, the cpu property ``sve-default-vector-length=N`` is 401defined to mirror the Linux kernel parameter file 402``/proc/sys/abi/sve_default_vector_length``. The default length, ``N``, 403is in units of bytes and must be between 16 and 8192. 404If not specified, the default vector length is 64. 405 406If the default length is larger than the maximum vector length enabled, 407the actual vector length will be reduced. Note that the maximum vector 408length supported by QEMU is 256. 409 410If this property is set to ``-1`` then the default vector length 411is set to the maximum possible length. 412 413SME CPU Properties 414================== 415 416The SME CPU properties are much like the SVE properties: ``sme`` is 417used to enable or disable the entire SME feature, and ``sme<N>`` is 418used to enable or disable specific vector lengths. Finally, 419``sme_fa64`` is used to enable or disable ``FEAT_SME_FA64``, which 420allows execution of the "full a64" instruction set while Streaming 421SVE mode is enabled. 422 423SME is not supported by KVM at this time. 424 425At least one vector length must be enabled when ``sme`` is enabled, 426and all vector lengths must be powers of 2. The maximum vector 427length supported by qemu is 2048 bits. Otherwise, there are no 428additional constraints on the set of vector lengths supported by SME. 429 430SME User-mode Default Vector Length Property 431-------------------------------------------- 432 433For qemu-aarch64, the cpu property ``sme-default-vector-length=N`` is 434defined to mirror the Linux kernel parameter file 435``/proc/sys/abi/sme_default_vector_length``. The default length, ``N``, 436is in units of bytes and must be between 16 and 8192. 437If not specified, the default vector length is 32. 438 439As with ``sve-default-vector-length``, if the default length is larger 440than the maximum vector length enabled, the actual vector length will 441be reduced. If this property is set to ``-1`` then the default vector 442length is set to the maximum possible length. 443 444RME CPU Properties 445================== 446 447The status of RME support with QEMU is experimental. At this time we 448only support RME within the CPU proper, not within the SMMU or GIC. 449The feature is enabled by the CPU property ``x-rme``, with the ``x-`` 450prefix present as a reminder of the experimental status, and defaults off. 451 452The method for enabling RME will change in some future QEMU release 453without notice or backward compatibility. 454 455RME Level 0 GPT Size Property 456----------------------------- 457 458To aid firmware developers in testing different possible CPU 459configurations, ``x-l0gptsz=S`` may be used to specify the value 460to encode into ``GPCCR_EL3.L0GPTSZ``, a read-only field that 461specifies the size of the Level 0 Granule Protection Table. 462Legal values for ``S`` are 30, 34, 36, and 39; the default is 30. 463 464As with ``x-rme``, the ``x-l0gptsz`` property may be renamed or 465removed in some future QEMU release. 466