xref: /openbmc/qemu/docs/system/i386/hyperv.rst (revision cde3c425)
1Hyper-V Enlightenments
2======================
3
4
5Description
6-----------
7
8In some cases when implementing a hardware interface in software is slow, KVM
9implements its own paravirtualized interfaces. This works well for Linux as
10guest support for such features is added simultaneously with the feature itself.
11It may, however, be hard-to-impossible to add support for these interfaces to
12proprietary OSes, namely, Microsoft Windows.
13
14KVM on x86 implements Hyper-V Enlightenments for Windows guests. These features
15make Windows and Hyper-V guests think they're running on top of a Hyper-V
16compatible hypervisor and use Hyper-V specific features.
17
18
19Setup
20-----
21
22No Hyper-V enlightenments are enabled by default by either KVM or QEMU. In
23QEMU, individual enlightenments can be enabled through CPU flags, e.g:
24
25.. parsed-literal::
26
27  |qemu_system| --enable-kvm --cpu host,hv_relaxed,hv_vpindex,hv_time, ...
28
29Sometimes there are dependencies between enlightenments, QEMU is supposed to
30check that the supplied configuration is sane.
31
32When any set of the Hyper-V enlightenments is enabled, QEMU changes hypervisor
33identification (CPUID 0x40000000..0x4000000A) to Hyper-V. KVM identification
34and features are kept in leaves 0x40000100..0x40000101.
35
36
37Existing enlightenments
38-----------------------
39
40``hv-relaxed``
41  This feature tells guest OS to disable watchdog timeouts as it is running on a
42  hypervisor. It is known that some Windows versions will do this even when they
43  see 'hypervisor' CPU flag.
44
45``hv-vapic``
46  Provides so-called VP Assist page MSR to guest allowing it to work with APIC
47  more efficiently. In particular, this enlightenment allows paravirtualized
48  (exit-less) EOI processing.
49
50``hv-spinlocks`` = xxx
51  Enables paravirtualized spinlocks. The parameter indicates how many times
52  spinlock acquisition should be attempted before indicating the situation to the
53  hypervisor. A special value 0xffffffff indicates "never notify".
54
55``hv-vpindex``
56  Provides HV_X64_MSR_VP_INDEX (0x40000002) MSR to the guest which has Virtual
57  processor index information. This enlightenment makes sense in conjunction with
58  hv-synic, hv-stimer and other enlightenments which require the guest to know its
59  Virtual Processor indices (e.g. when VP index needs to be passed in a
60  hypercall).
61
62``hv-runtime``
63  Provides HV_X64_MSR_VP_RUNTIME (0x40000010) MSR to the guest. The MSR keeps the
64  virtual processor run time in 100ns units. This gives guest operating system an
65  idea of how much time was 'stolen' from it (when the virtual CPU was preempted
66  to perform some other work).
67
68``hv-crash``
69  Provides HV_X64_MSR_CRASH_P0..HV_X64_MSR_CRASH_P5 (0x40000100..0x40000105) and
70  HV_X64_MSR_CRASH_CTL (0x40000105) MSRs to the guest. These MSRs are written to
71  by the guest when it crashes, HV_X64_MSR_CRASH_P0..HV_X64_MSR_CRASH_P5 MSRs
72  contain additional crash information. This information is outputted in QEMU log
73  and through QAPI.
74  Note: unlike under genuine Hyper-V, write to HV_X64_MSR_CRASH_CTL causes guest
75  to shutdown. This effectively blocks crash dump generation by Windows.
76
77``hv-time``
78  Enables two Hyper-V-specific clocksources available to the guest: MSR-based
79  Hyper-V clocksource (HV_X64_MSR_TIME_REF_COUNT, 0x40000020) and Reference TSC
80  page (enabled via MSR HV_X64_MSR_REFERENCE_TSC, 0x40000021). Both clocksources
81  are per-guest, Reference TSC page clocksource allows for exit-less time stamp
82  readings. Using this enlightenment leads to significant speedup of all timestamp
83  related operations.
84
85``hv-synic``
86  Enables Hyper-V Synthetic interrupt controller - an extension of a local APIC.
87  When enabled, this enlightenment provides additional communication facilities
88  to the guest: SynIC messages and Events. This is a pre-requisite for
89  implementing VMBus devices (not yet in QEMU). Additionally, this enlightenment
90  is needed to enable Hyper-V synthetic timers. SynIC is controlled through MSRs
91  HV_X64_MSR_SCONTROL..HV_X64_MSR_EOM (0x40000080..0x40000084) and
92  HV_X64_MSR_SINT0..HV_X64_MSR_SINT15 (0x40000090..0x4000009F)
93
94  Requires: ``hv-vpindex``
95
96``hv-stimer``
97  Enables Hyper-V synthetic timers. There are four synthetic timers per virtual
98  CPU controlled through HV_X64_MSR_STIMER0_CONFIG..HV_X64_MSR_STIMER3_COUNT
99  (0x400000B0..0x400000B7) MSRs. These timers can work either in single-shot or
100  periodic mode. It is known that certain Windows versions revert to using HPET
101  (or even RTC when HPET is unavailable) extensively when this enlightenment is
102  not provided; this can lead to significant CPU consumption, even when virtual
103  CPU is idle.
104
105  Requires: ``hv-vpindex``, ``hv-synic``, ``hv-time``
106
107``hv-tlbflush``
108  Enables paravirtualized TLB shoot-down mechanism. On x86 architecture, remote
109  TLB flush procedure requires sending IPIs and waiting for other CPUs to perform
110  local TLB flush. In virtualized environment some virtual CPUs may not even be
111  scheduled at the time of the call and may not require flushing (or, flushing
112  may be postponed until the virtual CPU is scheduled). hv-tlbflush enlightenment
113  implements TLB shoot-down through hypervisor enabling the optimization.
114
115  Requires: ``hv-vpindex``
116
117``hv-ipi``
118  Enables paravirtualized IPI send mechanism. HvCallSendSyntheticClusterIpi
119  hypercall may target more than 64 virtual CPUs simultaneously, doing the same
120  through APIC requires more than one access (and thus exit to the hypervisor).
121
122  Requires: ``hv-vpindex``
123
124``hv-vendor-id`` = xxx
125  This changes Hyper-V identification in CPUID 0x40000000.EBX-EDX from the default
126  "Microsoft Hv". The parameter should be no longer than 12 characters. According
127  to the specification, guests shouldn't use this information and it is unknown
128  if there is a Windows version which acts differently.
129  Note: hv-vendor-id is not an enlightenment and thus doesn't enable Hyper-V
130  identification when specified without some other enlightenment.
131
132``hv-reset``
133  Provides HV_X64_MSR_RESET (0x40000003) MSR to the guest allowing it to reset
134  itself by writing to it. Even when this MSR is enabled, it is not a recommended
135  way for Windows to perform system reboot and thus it may not be used.
136
137``hv-frequencies``
138  Provides HV_X64_MSR_TSC_FREQUENCY (0x40000022) and HV_X64_MSR_APIC_FREQUENCY
139  (0x40000023) allowing the guest to get its TSC/APIC frequencies without doing
140  measurements.
141
142``hv-reenlightenment``
143  The enlightenment is nested specific, it targets Hyper-V on KVM guests. When
144  enabled, it provides HV_X64_MSR_REENLIGHTENMENT_CONTROL (0x40000106),
145  HV_X64_MSR_TSC_EMULATION_CONTROL (0x40000107)and HV_X64_MSR_TSC_EMULATION_STATUS
146  (0x40000108) MSRs allowing the guest to get notified when TSC frequency changes
147  (only happens on migration) and keep using old frequency (through emulation in
148  the hypervisor) until it is ready to switch to the new one. This, in conjunction
149  with ``hv-frequencies``, allows Hyper-V on KVM to pass stable clocksource
150  (Reference TSC page) to its own guests.
151
152  Note, KVM doesn't fully support re-enlightenment notifications and doesn't
153  emulate TSC accesses after migration so 'tsc-frequency=' CPU option also has to
154  be specified to make migration succeed. The destination host has to either have
155  the same TSC frequency or support TSC scaling CPU feature.
156
157  Recommended: ``hv-frequencies``
158
159``hv-evmcs``
160  The enlightenment is nested specific, it targets Hyper-V on KVM guests. When
161  enabled, it provides Enlightened VMCS version 1 feature to the guest. The feature
162  implements paravirtualized protocol between L0 (KVM) and L1 (Hyper-V)
163  hypervisors making L2 exits to the hypervisor faster. The feature is Intel-only.
164
165  Note: some virtualization features (e.g. Posted Interrupts) are disabled when
166  hv-evmcs is enabled. It may make sense to measure your nested workload with and
167  without the feature to find out if enabling it is beneficial.
168
169  Requires: ``hv-vapic``
170
171``hv-stimer-direct``
172  Hyper-V specification allows synthetic timer operation in two modes: "classic",
173  when expiration event is delivered as SynIC message and "direct", when the event
174  is delivered via normal interrupt. It is known that nested Hyper-V can only
175  use synthetic timers in direct mode and thus ``hv-stimer-direct`` needs to be
176  enabled.
177
178  Requires: ``hv-vpindex``, ``hv-synic``, ``hv-time``, ``hv-stimer``
179
180``hv-avic`` (``hv-apicv``)
181  The enlightenment allows to use Hyper-V SynIC with hardware APICv/AVIC enabled.
182  Normally, Hyper-V SynIC disables these hardware feature and suggests the guest
183  to use paravirtualized AutoEOI feature.
184  Note: enabling this feature on old hardware (without APICv/AVIC support) may
185  have negative effect on guest's performance.
186
187``hv-no-nonarch-coresharing`` = on/off/auto
188  This enlightenment tells guest OS that virtual processors will never share a
189  physical core unless they are reported as sibling SMT threads. This information
190  is required by Windows and Hyper-V guests to properly mitigate SMT related CPU
191  vulnerabilities.
192
193  When the option is set to 'auto' QEMU will enable the feature only when KVM
194  reports that non-architectural coresharing is impossible, this means that
195  hyper-threading is not supported or completely disabled on the host. This
196  setting also prevents migration as SMT settings on the destination may differ.
197  When the option is set to 'on' QEMU will always enable the feature, regardless
198  of host setup. To keep guests secure, this can only be used in conjunction with
199  exposing correct vCPU topology and vCPU pinning.
200
201``hv-version-id-build``, ``hv-version-id-major``, ``hv-version-id-minor``, ``hv-version-id-spack``, ``hv-version-id-sbranch``, ``hv-version-id-snumber``
202  This changes Hyper-V version identification in CPUID 0x40000002.EAX-EDX from the
203  default (WS2016).
204
205  - ``hv-version-id-build`` sets 'Build Number' (32 bits)
206  - ``hv-version-id-major`` sets 'Major Version' (16 bits)
207  - ``hv-version-id-minor`` sets 'Minor Version' (16 bits)
208  - ``hv-version-id-spack`` sets 'Service Pack' (32 bits)
209  - ``hv-version-id-sbranch`` sets 'Service Branch' (8 bits)
210  - ``hv-version-id-snumber`` sets 'Service Number' (24 bits)
211
212  Note: hv-version-id-* are not enlightenments and thus don't enable Hyper-V
213  identification when specified without any other enlightenments.
214
215``hv-syndbg``
216  Enables Hyper-V synthetic debugger interface, this is a special interface used
217  by Windows Kernel debugger to send the packets through, rather than sending
218  them via serial/network .
219  When enabled, this enlightenment provides additional communication facilities
220  to the guest: SynDbg messages.
221  This new communication is used by Windows Kernel debugger rather than sending
222  packets via serial/network, adding significant performance boost over the other
223  comm channels.
224  This enlightenment requires a VMBus device (-device vmbus-bridge,irq=15).
225
226  Requires: ``hv-relaxed``, ``hv_time``, ``hv-vapic``, ``hv-vpindex``, ``hv-synic``, ``hv-runtime``, ``hv-stimer``
227
228``hv-emsr-bitmap``
229  The enlightenment is nested specific, it targets Hyper-V on KVM guests. When
230  enabled, it allows L0 (KVM) and L1 (Hyper-V) hypervisors to collaborate to
231  avoid unnecessary updates to L2 MSR-Bitmap upon vmexits. While the protocol is
232  supported for both VMX (Intel) and SVM (AMD), the VMX implementation requires
233  Enlightened VMCS (``hv-evmcs``) feature to also be enabled.
234
235  Recommended: ``hv-evmcs`` (Intel)
236
237``hv-xmm-input``
238  Hyper-V specification allows to pass parameters for certain hypercalls using XMM
239  registers ("XMM Fast Hypercall Input"). When the feature is in use, it allows
240  for faster hypercalls processing as KVM can avoid reading guest's memory.
241
242``hv-tlbflush-ext``
243  Allow for extended GVA ranges to be passed to Hyper-V TLB flush hypercalls
244  (HvFlushVirtualAddressList/HvFlushVirtualAddressListEx).
245
246  Requires: ``hv-tlbflush``
247
248``hv-tlbflush-direct``
249  The enlightenment is nested specific, it targets Hyper-V on KVM guests. When
250  enabled, it allows L0 (KVM) to directly handle TLB flush hypercalls from L2
251  guest without the need to exit to L1 (Hyper-V) hypervisor. While the feature is
252  supported for both VMX (Intel) and SVM (AMD), the VMX implementation requires
253  Enlightened VMCS (``hv-evmcs``) feature to also be enabled.
254
255  Requires: ``hv-vapic``
256
257  Recommended: ``hv-evmcs`` (Intel)
258
259Supplementary features
260----------------------
261
262``hv-passthrough``
263  In some cases (e.g. during development) it may make sense to use QEMU in
264  'pass-through' mode and give Windows guests all enlightenments currently
265  supported by KVM.
266
267  Note: ``hv-passthrough`` flag only enables enlightenments which are known to QEMU
268  (have corresponding 'hv-' flag) and copies ``hv-spinlocks`` and ``hv-vendor-id``
269  values from KVM to QEMU. ``hv-passthrough`` overrides all other 'hv-' settings on
270  the command line.
271
272  Note: ``hv-passthrough`` does not enable ``hv-syndbg`` which can prevent certain
273  Windows guests from booting when used without proper configuration. If needed,
274  ``hv-syndbg`` can be enabled additionally.
275
276  Note: ``hv-passthrough`` effectively prevents migration as the list of enabled
277  enlightenments may differ between target and destination hosts.
278
279``hv-enforce-cpuid``
280  By default, KVM allows the guest to use all currently supported Hyper-V
281  enlightenments when Hyper-V CPUID interface was exposed, regardless of if
282  some features were not announced in guest visible CPUIDs. ``hv-enforce-cpuid``
283  feature alters this behavior and only allows the guest to use exposed Hyper-V
284  enlightenments.
285
286Recommendations
287---------------
288
289To achieve the best performance of Windows and Hyper-V guests and unless there
290are any specific requirements (e.g. migration to older QEMU/KVM versions,
291emulating specific Hyper-V version, ...), it is recommended to enable all
292currently implemented Hyper-V enlightenments with the following exceptions:
293
294- ``hv-syndbg``, ``hv-passthrough``, ``hv-enforce-cpuid`` should not be enabled
295  in production configurations as these are debugging/development features.
296- ``hv-reset`` can be avoided as modern Hyper-V versions don't expose it.
297- ``hv-evmcs`` can (and should) be enabled on Intel CPUs only. While the feature
298  is only used in nested configurations (Hyper-V, WSL2), enabling it for regular
299  Windows guests should not have any negative effects.
300- ``hv-no-nonarch-coresharing`` must only be enabled if vCPUs are properly pinned
301  so no non-architectural core sharing is possible.
302- ``hv-vendor-id``, ``hv-version-id-build``, ``hv-version-id-major``,
303  ``hv-version-id-minor``, ``hv-version-id-spack``, ``hv-version-id-sbranch``,
304  ``hv-version-id-snumber`` can be left unchanged, guests are not supposed to
305  behave differently when different Hyper-V version is presented to them.
306- ``hv-crash`` must only be enabled if the crash information is consumed via
307  QAPI by higher levels of the virtualization stack. Enabling this feature
308  effectively prevents Windows from creating dumps upon crashes.
309- ``hv-reenlightenment`` can only be used on hardware which supports TSC
310  scaling or when guest migration is not needed.
311- ``hv-spinlocks`` should be set to e.g. 0xfff when host CPUs are overcommited
312  (meaning there are other scheduled tasks or guests) and can be left unchanged
313  from the default value (0xffffffff) otherwise.
314- ``hv-avic``/``hv-apicv`` should not be enabled if the hardware does not
315  support APIC virtualization (Intel APICv, AMD AVIC).
316
317Useful links
318------------
319Hyper-V Top Level Functional specification and other information:
320
321- https://github.com/MicrosoftDocs/Virtualization-Documentation
322- https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/tlfs
323
324