1Recommendations for KVM CPU model configuration on x86 hosts
2~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3
4The information that follows provides recommendations for configuring
5CPU models on x86 hosts. The goals are to maximise performance, while
6protecting guest OS against various CPU hardware flaws, and optionally
7enabling live migration between hosts with heterogeneous CPU models.
8
9
10Two ways to configure CPU models with QEMU / KVM
11^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
12
13(1) **Host passthrough**
14
15    This passes the host CPU model features, model, stepping, exactly to
16    the guest. Note that KVM may filter out some host CPU model features
17    if they cannot be supported with virtualization. Live migration is
18    unsafe when this mode is used as libvirt / QEMU cannot guarantee a
19    stable CPU is exposed to the guest across hosts. This is the
20    recommended CPU to use, provided live migration is not required.
21
22(2) **Named model**
23
24    QEMU comes with a number of predefined named CPU models, that
25    typically refer to specific generations of hardware released by
26    Intel and AMD.  These allow the guest VMs to have a degree of
27    isolation from the host CPU, allowing greater flexibility in live
28    migrating between hosts with differing hardware.  @end table
29
30In both cases, it is possible to optionally add or remove individual CPU
31features, to alter what is presented to the guest by default.
32
33Libvirt supports a third way to configure CPU models known as "Host
34model".  This uses the QEMU "Named model" feature, automatically picking
35a CPU model that is similar the host CPU, and then adding extra features
36to approximate the host model as closely as possible. This does not
37guarantee the CPU family, stepping, etc will precisely match the host
38CPU, as they would with "Host passthrough", but gives much of the
39benefit of passthrough, while making live migration safe.
40
41
42ABI compatibility levels for CPU models
43^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
44
45The x86_64 architecture has a number of `ABI compatibility levels`_
46defined. Traditionally most operating systems and toolchains would
47only target the original baseline ABI. It is expected that in
48future OS and toolchains are likely to target newer ABIs. The
49table that follows illustrates which ABI compatibility levels
50can be satisfied by the QEMU CPU models. Note that the table only
51lists the long term stable CPU model versions (eg Haswell-v4).
52In addition to whats listed, there are also many CPU model
53aliases which resolve to a different CPU model version,
54depending on the machine type is in use.
55
56.. _ABI compatibility levels: https://gitlab.com/x86-psABIs/x86-64-ABI/
57
58.. csv-table:: x86-64 ABI compatibility levels
59   :file: cpu-models-x86-abi.csv
60   :widths: 40,15,15,15,15
61   :header-rows: 2
62
63
64Preferred CPU models for Intel x86 hosts
65^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
66
67The following CPU models are preferred for use on Intel hosts.
68Administrators / applications are recommended to use the CPU model that
69matches the generation of the host CPUs in use. In a deployment with a
70mixture of host CPU models between machines, if live migration
71compatibility is required, use the newest CPU model that is compatible
72across all desired hosts.
73
74``Cascadelake-Server``, ``Cascadelake-Server-noTSX``
75    Intel Xeon Processor (Cascade Lake, 2019), with "stepping" levels 6
76    or 7 only.  (The Cascade Lake Xeon processor with *stepping 5 is
77    vulnerable to MDS variants*.)
78
79``Skylake-Server``, ``Skylake-Server-IBRS``, ``Skylake-Server-IBRS-noTSX``
80    Intel Xeon Processor (Skylake, 2016)
81
82``Skylake-Client``, ``Skylake-Client-IBRS``, ``Skylake-Client-noTSX-IBRS}``
83    Intel Core Processor (Skylake, 2015)
84
85``Broadwell``, ``Broadwell-IBRS``, ``Broadwell-noTSX``, ``Broadwell-noTSX-IBRS``
86    Intel Core Processor (Broadwell, 2014)
87
88``Haswell``, ``Haswell-IBRS``, ``Haswell-noTSX``, ``Haswell-noTSX-IBRS``
89    Intel Core Processor (Haswell, 2013)
90
91``IvyBridge``, ``IvyBridge-IBR``
92    Intel Xeon E3-12xx v2 (Ivy Bridge, 2012)
93
94``SandyBridge``, ``SandyBridge-IBRS``
95    Intel Xeon E312xx (Sandy Bridge, 2011)
96
97``Westmere``, ``Westmere-IBRS``
98    Westmere E56xx/L56xx/X56xx (Nehalem-C, 2010)
99
100``Nehalem``, ``Nehalem-IBRS``
101    Intel Core i7 9xx (Nehalem Class Core i7, 2008)
102
103``Penryn``
104    Intel Core 2 Duo P9xxx (Penryn Class Core 2, 2007)
105
106``Conroe``
107    Intel Celeron_4x0 (Conroe/Merom Class Core 2, 2006)
108
109
110Important CPU features for Intel x86 hosts
111^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
112
113The following are important CPU features that should be used on Intel
114x86 hosts, when available in the host CPU. Some of them require explicit
115configuration to enable, as they are not included by default in some, or
116all, of the named CPU models listed above. In general all of these
117features are included if using "Host passthrough" or "Host model".
118
119``pcid``
120  Recommended to mitigate the cost of the Meltdown (CVE-2017-5754) fix.
121
122  Included by default in Haswell, Broadwell & Skylake Intel CPU models.
123
124  Should be explicitly turned on for Westmere, SandyBridge, and
125  IvyBridge Intel CPU models. Note that some desktop/mobile Westmere
126  CPUs cannot support this feature.
127
128``spec-ctrl``
129  Required to enable the Spectre v2 (CVE-2017-5715) fix.
130
131  Included by default in Intel CPU models with -IBRS suffix.
132
133  Must be explicitly turned on for Intel CPU models without -IBRS
134  suffix.
135
136  Requires the host CPU microcode to support this feature before it
137  can be used for guest CPUs.
138
139``stibp``
140  Required to enable stronger Spectre v2 (CVE-2017-5715) fixes in some
141  operating systems.
142
143  Must be explicitly turned on for all Intel CPU models.
144
145  Requires the host CPU microcode to support this feature before it can
146  be used for guest CPUs.
147
148``ssbd``
149  Required to enable the CVE-2018-3639 fix.
150
151  Not included by default in any Intel CPU model.
152
153  Must be explicitly turned on for all Intel CPU models.
154
155  Requires the host CPU microcode to support this feature before it
156  can be used for guest CPUs.
157
158``pdpe1gb``
159  Recommended to allow guest OS to use 1GB size pages.
160
161  Not included by default in any Intel CPU model.
162
163  Should be explicitly turned on for all Intel CPU models.
164
165  Note that not all CPU hardware will support this feature.
166
167``md-clear``
168  Required to confirm the MDS (CVE-2018-12126, CVE-2018-12127,
169  CVE-2018-12130, CVE-2019-11091) fixes.
170
171  Not included by default in any Intel CPU model.
172
173  Must be explicitly turned on for all Intel CPU models.
174
175  Requires the host CPU microcode to support this feature before it
176  can be used for guest CPUs.
177
178``mds-no``
179  Recommended to inform the guest OS that the host is *not* vulnerable
180  to any of the MDS variants ([MFBDS] CVE-2018-12130, [MLPDS]
181  CVE-2018-12127, [MSBDS] CVE-2018-12126).
182
183  This is an MSR (Model-Specific Register) feature rather than a CPUID feature,
184  so it will not appear in the Linux ``/proc/cpuinfo`` in the host or
185  guest.  Instead, the host kernel uses it to populate the MDS
186  vulnerability file in ``sysfs``.
187
188  So it should only be enabled for VMs if the host reports @code{Not
189  affected} in the ``/sys/devices/system/cpu/vulnerabilities/mds`` file.
190
191``taa-no``
192  Recommended to inform that the guest that the host is ``not``
193  vulnerable to CVE-2019-11135, TSX Asynchronous Abort (TAA).
194
195  This too is an MSR feature, so it does not show up in the Linux
196  ``/proc/cpuinfo`` in the host or guest.
197
198  It should only be enabled for VMs if the host reports ``Not affected``
199  in the ``/sys/devices/system/cpu/vulnerabilities/tsx_async_abort``
200  file.
201
202``tsx-ctrl``
203  Recommended to inform the guest that it can disable the Intel TSX
204  (Transactional Synchronization Extensions) feature; or, if the
205  processor is vulnerable, use the Intel VERW instruction (a
206  processor-level instruction that performs checks on memory access) as
207  a mitigation for the TAA vulnerability.  (For details, refer to
208  Intel's `deep dive into MDS
209  <https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling>`_.)
210
211  Expose this to the guest OS if and only if: (a) the host has TSX
212  enabled; *and* (b) the guest has ``rtm`` CPU flag enabled.
213
214  By disabling TSX, KVM-based guests can avoid paying the price of
215  mitigating TSX-based attacks.
216
217  Note that ``tsx-ctrl`` too is an MSR feature, so it does not show
218  up in the Linux ``/proc/cpuinfo`` in the host or guest.
219
220  To validate that Intel TSX is indeed disabled for the guest, there are
221  two ways: (a) check for the *absence* of ``rtm`` in the guest's
222  ``/proc/cpuinfo``; or (b) the
223  ``/sys/devices/system/cpu/vulnerabilities/tsx_async_abort`` file in
224  the guest should report ``Mitigation: TSX disabled``.
225
226
227Preferred CPU models for AMD x86 hosts
228^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
229
230The following CPU models are preferred for use on Intel hosts.
231Administrators / applications are recommended to use the CPU model that
232matches the generation of the host CPUs in use. In a deployment with a
233mixture of host CPU models between machines, if live migration
234compatibility is required, use the newest CPU model that is compatible
235across all desired hosts.
236
237``EPYC``, ``EPYC-IBPB``
238    AMD EPYC Processor (2017)
239
240``Opteron_G5``
241    AMD Opteron 63xx class CPU (2012)
242
243``Opteron_G4``
244    AMD Opteron 62xx class CPU (2011)
245
246``Opteron_G3``
247    AMD Opteron 23xx (Gen 3 Class Opteron, 2009)
248
249``Opteron_G2``
250    AMD Opteron 22xx (Gen 2 Class Opteron, 2006)
251
252``Opteron_G1``
253    AMD Opteron 240 (Gen 1 Class Opteron, 2004)
254
255
256Important CPU features for AMD x86 hosts
257^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
258
259The following are important CPU features that should be used on AMD x86
260hosts, when available in the host CPU. Some of them require explicit
261configuration to enable, as they are not included by default in some, or
262all, of the named CPU models listed above. In general all of these
263features are included if using "Host passthrough" or "Host model".
264
265``ibpb``
266  Required to enable the Spectre v2 (CVE-2017-5715) fix.
267
268  Included by default in AMD CPU models with -IBPB suffix.
269
270  Must be explicitly turned on for AMD CPU models without -IBPB suffix.
271
272  Requires the host CPU microcode to support this feature before it
273  can be used for guest CPUs.
274
275``stibp``
276  Required to enable stronger Spectre v2 (CVE-2017-5715) fixes in some
277  operating systems.
278
279  Must be explicitly turned on for all AMD CPU models.
280
281  Requires the host CPU microcode to support this feature before it
282  can be used for guest CPUs.
283
284``virt-ssbd``
285  Required to enable the CVE-2018-3639 fix
286
287  Not included by default in any AMD CPU model.
288
289  Must be explicitly turned on for all AMD CPU models.
290
291  This should be provided to guests, even if amd-ssbd is also provided,
292  for maximum guest compatibility.
293
294  Note for some QEMU / libvirt versions, this must be force enabled when
295  when using "Host model", because this is a virtual feature that
296  doesn't exist in the physical host CPUs.
297
298``amd-ssbd``
299  Required to enable the CVE-2018-3639 fix
300
301  Not included by default in any AMD CPU model.
302
303  Must be explicitly turned on for all AMD CPU models.
304
305  This provides higher performance than ``virt-ssbd`` so should be
306  exposed to guests whenever available in the host. ``virt-ssbd`` should
307  none the less also be exposed for maximum guest compatibility as some
308  kernels only know about ``virt-ssbd``.
309
310``amd-no-ssb``
311  Recommended to indicate the host is not vulnerable CVE-2018-3639
312
313  Not included by default in any AMD CPU model.
314
315  Future hardware generations of CPU will not be vulnerable to
316  CVE-2018-3639, and thus the guest should be told not to enable
317  its mitigations, by exposing amd-no-ssb. This is mutually
318  exclusive with virt-ssbd and amd-ssbd.
319
320``pdpe1gb``
321  Recommended to allow guest OS to use 1GB size pages
322
323  Not included by default in any AMD CPU model.
324
325  Should be explicitly turned on for all AMD CPU models.
326
327  Note that not all CPU hardware will support this feature.
328
329
330Default x86 CPU models
331^^^^^^^^^^^^^^^^^^^^^^
332
333The default QEMU CPU models are designed such that they can run on all
334hosts.  If an application does not wish to do perform any host
335compatibility checks before launching guests, the default is guaranteed
336to work.
337
338The default CPU models will, however, leave the guest OS vulnerable to
339various CPU hardware flaws, so their use is strongly discouraged.
340Applications should follow the earlier guidance to setup a better CPU
341configuration, with host passthrough recommended if live migration is
342not needed.
343
344``qemu32``, ``qemu64``
345    QEMU Virtual CPU version 2.5+ (32 & 64 bit variants)
346
347``qemu64`` is used for x86_64 guests and ``qemu32`` is used for i686
348guests, when no ``-cpu`` argument is given to QEMU, or no ``<cpu>`` is
349provided in libvirt XML.
350
351Other non-recommended x86 CPUs
352^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
353
354The following CPUs models are compatible with most AMD and Intel x86
355hosts, but their usage is discouraged, as they expose a very limited
356featureset, which prevents guests having optimal performance.
357
358``kvm32``, ``kvm64``
359    Common KVM processor (32 & 64 bit variants).
360
361    Legacy models just for historical compatibility with ancient QEMU
362    versions.
363
364``486``, ``athlon``, ``phenom``, ``coreduo``, ``core2duo``, ``n270``, ``pentium``, ``pentium2``, ``pentium3``
365    Various very old x86 CPU models, mostly predating the introduction
366    of hardware assisted virtualization, that should thus not be
367    required for running virtual machines.
368
369
370Syntax for configuring CPU models
371~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
372
373The examples below illustrate the approach to configuring the various
374CPU models / features in QEMU and libvirt.
375
376QEMU command line
377^^^^^^^^^^^^^^^^^
378
379Host passthrough:
380
381.. parsed-literal::
382
383  |qemu_system| -cpu host
384
385Host passthrough with feature customization:
386
387.. parsed-literal::
388
389  |qemu_system| -cpu host,vmx=off,...
390
391Named CPU models:
392
393.. parsed-literal::
394
395  |qemu_system| -cpu Westmere
396
397Named CPU models with feature customization:
398
399.. parsed-literal::
400
401  |qemu_system| -cpu Westmere,pcid=on,...
402
403Libvirt guest XML
404^^^^^^^^^^^^^^^^^
405
406Host passthrough::
407
408    <cpu mode='host-passthrough'/>
409
410Host passthrough with feature customization::
411
412    <cpu mode='host-passthrough'>
413        <feature name="vmx" policy="disable"/>
414        ...
415    </cpu>
416
417Host model::
418
419    <cpu mode='host-model'/>
420
421Host model with feature customization::
422
423    <cpu mode='host-model'>
424        <feature name="vmx" policy="disable"/>
425        ...
426    </cpu>
427
428Named model::
429
430    <cpu mode='custom'>
431        <model name="Westmere"/>
432    </cpu>
433
434Named model with feature customization::
435
436    <cpu mode='custom'>
437        <model name="Westmere"/>
438        <feature name="pcid" policy="require"/>
439        ...
440    </cpu>
441