1Hyper-V Enlightenments 2====================== 3 4 5Description 6----------- 7 8In some cases when implementing a hardware interface in software is slow, KVM 9implements its own paravirtualized interfaces. This works well for Linux as 10guest support for such features is added simultaneously with the feature itself. 11It may, however, be hard-to-impossible to add support for these interfaces to 12proprietary OSes, namely, Microsoft Windows. 13 14KVM on x86 implements Hyper-V Enlightenments for Windows guests. These features 15make Windows and Hyper-V guests think they're running on top of a Hyper-V 16compatible hypervisor and use Hyper-V specific features. 17 18 19Setup 20----- 21 22No Hyper-V enlightenments are enabled by default by either KVM or QEMU. In 23QEMU, individual enlightenments can be enabled through CPU flags, e.g: 24 25.. parsed-literal:: 26 27 |qemu_system| --enable-kvm --cpu host,hv_relaxed,hv_vpindex,hv_time, ... 28 29Sometimes there are dependencies between enlightenments, QEMU is supposed to 30check that the supplied configuration is sane. 31 32When any set of the Hyper-V enlightenments is enabled, QEMU changes hypervisor 33identification (CPUID 0x40000000..0x4000000A) to Hyper-V. KVM identification 34and features are kept in leaves 0x40000100..0x40000101. 35 36 37Existing enlightenments 38----------------------- 39 40``hv-relaxed`` 41 This feature tells guest OS to disable watchdog timeouts as it is running on a 42 hypervisor. It is known that some Windows versions will do this even when they 43 see 'hypervisor' CPU flag. 44 45``hv-vapic`` 46 Provides so-called VP Assist page MSR to guest allowing it to work with APIC 47 more efficiently. In particular, this enlightenment allows paravirtualized 48 (exit-less) EOI processing. 49 50``hv-spinlocks`` = xxx 51 Enables paravirtualized spinlocks. The parameter indicates how many times 52 spinlock acquisition should be attempted before indicating the situation to the 53 hypervisor. A special value 0xffffffff indicates "never notify". 54 55``hv-vpindex`` 56 Provides HV_X64_MSR_VP_INDEX (0x40000002) MSR to the guest which has Virtual 57 processor index information. This enlightenment makes sense in conjunction with 58 hv-synic, hv-stimer and other enlightenments which require the guest to know its 59 Virtual Processor indices (e.g. when VP index needs to be passed in a 60 hypercall). 61 62``hv-runtime`` 63 Provides HV_X64_MSR_VP_RUNTIME (0x40000010) MSR to the guest. The MSR keeps the 64 virtual processor run time in 100ns units. This gives guest operating system an 65 idea of how much time was 'stolen' from it (when the virtual CPU was preempted 66 to perform some other work). 67 68``hv-crash`` 69 Provides HV_X64_MSR_CRASH_P0..HV_X64_MSR_CRASH_P5 (0x40000100..0x40000105) and 70 HV_X64_MSR_CRASH_CTL (0x40000105) MSRs to the guest. These MSRs are written to 71 by the guest when it crashes, HV_X64_MSR_CRASH_P0..HV_X64_MSR_CRASH_P5 MSRs 72 contain additional crash information. This information is outputted in QEMU log 73 and through QAPI. 74 Note: unlike under genuine Hyper-V, write to HV_X64_MSR_CRASH_CTL causes guest 75 to shutdown. This effectively blocks crash dump generation by Windows. 76 77``hv-time`` 78 Enables two Hyper-V-specific clocksources available to the guest: MSR-based 79 Hyper-V clocksource (HV_X64_MSR_TIME_REF_COUNT, 0x40000020) and Reference TSC 80 page (enabled via MSR HV_X64_MSR_REFERENCE_TSC, 0x40000021). Both clocksources 81 are per-guest, Reference TSC page clocksource allows for exit-less time stamp 82 readings. Using this enlightenment leads to significant speedup of all timestamp 83 related operations. 84 85``hv-synic`` 86 Enables Hyper-V Synthetic interrupt controller - an extension of a local APIC. 87 When enabled, this enlightenment provides additional communication facilities 88 to the guest: SynIC messages and Events. This is a pre-requisite for 89 implementing VMBus devices (not yet in QEMU). Additionally, this enlightenment 90 is needed to enable Hyper-V synthetic timers. SynIC is controlled through MSRs 91 HV_X64_MSR_SCONTROL..HV_X64_MSR_EOM (0x40000080..0x40000084) and 92 HV_X64_MSR_SINT0..HV_X64_MSR_SINT15 (0x40000090..0x4000009F) 93 94 Requires: ``hv-vpindex`` 95 96``hv-stimer`` 97 Enables Hyper-V synthetic timers. There are four synthetic timers per virtual 98 CPU controlled through HV_X64_MSR_STIMER0_CONFIG..HV_X64_MSR_STIMER3_COUNT 99 (0x400000B0..0x400000B7) MSRs. These timers can work either in single-shot or 100 periodic mode. It is known that certain Windows versions revert to using HPET 101 (or even RTC when HPET is unavailable) extensively when this enlightenment is 102 not provided; this can lead to significant CPU consumption, even when virtual 103 CPU is idle. 104 105 Requires: ``hv-vpindex``, ``hv-synic``, ``hv-time`` 106 107``hv-tlbflush`` 108 Enables paravirtualized TLB shoot-down mechanism. On x86 architecture, remote 109 TLB flush procedure requires sending IPIs and waiting for other CPUs to perform 110 local TLB flush. In virtualized environment some virtual CPUs may not even be 111 scheduled at the time of the call and may not require flushing (or, flushing 112 may be postponed until the virtual CPU is scheduled). hv-tlbflush enlightenment 113 implements TLB shoot-down through hypervisor enabling the optimization. 114 115 Requires: ``hv-vpindex`` 116 117``hv-ipi`` 118 Enables paravirtualized IPI send mechanism. HvCallSendSyntheticClusterIpi 119 hypercall may target more than 64 virtual CPUs simultaneously, doing the same 120 through APIC requires more than one access (and thus exit to the hypervisor). 121 122 Requires: ``hv-vpindex`` 123 124``hv-vendor-id`` = xxx 125 This changes Hyper-V identification in CPUID 0x40000000.EBX-EDX from the default 126 "Microsoft Hv". The parameter should be no longer than 12 characters. According 127 to the specification, guests shouldn't use this information and it is unknown 128 if there is a Windows version which acts differently. 129 Note: hv-vendor-id is not an enlightenment and thus doesn't enable Hyper-V 130 identification when specified without some other enlightenment. 131 132``hv-reset`` 133 Provides HV_X64_MSR_RESET (0x40000003) MSR to the guest allowing it to reset 134 itself by writing to it. Even when this MSR is enabled, it is not a recommended 135 way for Windows to perform system reboot and thus it may not be used. 136 137``hv-frequencies`` 138 Provides HV_X64_MSR_TSC_FREQUENCY (0x40000022) and HV_X64_MSR_APIC_FREQUENCY 139 (0x40000023) allowing the guest to get its TSC/APIC frequencies without doing 140 measurements. 141 142``hv-reenlightenment`` 143 The enlightenment is nested specific, it targets Hyper-V on KVM guests. When 144 enabled, it provides HV_X64_MSR_REENLIGHTENMENT_CONTROL (0x40000106), 145 HV_X64_MSR_TSC_EMULATION_CONTROL (0x40000107)and HV_X64_MSR_TSC_EMULATION_STATUS 146 (0x40000108) MSRs allowing the guest to get notified when TSC frequency changes 147 (only happens on migration) and keep using old frequency (through emulation in 148 the hypervisor) until it is ready to switch to the new one. This, in conjunction 149 with ``hv-frequencies``, allows Hyper-V on KVM to pass stable clocksource 150 (Reference TSC page) to its own guests. 151 152 Note, KVM doesn't fully support re-enlightenment notifications and doesn't 153 emulate TSC accesses after migration so 'tsc-frequency=' CPU option also has to 154 be specified to make migration succeed. The destination host has to either have 155 the same TSC frequency or support TSC scaling CPU feature. 156 157 Recommended: ``hv-frequencies`` 158 159``hv-evmcs`` 160 The enlightenment is nested specific, it targets Hyper-V on KVM guests. When 161 enabled, it provides Enlightened VMCS version 1 feature to the guest. The feature 162 implements paravirtualized protocol between L0 (KVM) and L1 (Hyper-V) 163 hypervisors making L2 exits to the hypervisor faster. The feature is Intel-only. 164 165 Note: some virtualization features (e.g. Posted Interrupts) are disabled when 166 hv-evmcs is enabled. It may make sense to measure your nested workload with and 167 without the feature to find out if enabling it is beneficial. 168 169 Requires: ``hv-vapic`` 170 171``hv-stimer-direct`` 172 Hyper-V specification allows synthetic timer operation in two modes: "classic", 173 when expiration event is delivered as SynIC message and "direct", when the event 174 is delivered via normal interrupt. It is known that nested Hyper-V can only 175 use synthetic timers in direct mode and thus ``hv-stimer-direct`` needs to be 176 enabled. 177 178 Requires: ``hv-vpindex``, ``hv-synic``, ``hv-time``, ``hv-stimer`` 179 180``hv-avic`` (``hv-apicv``) 181 The enlightenment allows to use Hyper-V SynIC with hardware APICv/AVIC enabled. 182 Normally, Hyper-V SynIC disables these hardware feature and suggests the guest 183 to use paravirtualized AutoEOI feature. 184 Note: enabling this feature on old hardware (without APICv/AVIC support) may 185 have negative effect on guest's performance. 186 187``hv-no-nonarch-coresharing`` = on/off/auto 188 This enlightenment tells guest OS that virtual processors will never share a 189 physical core unless they are reported as sibling SMT threads. This information 190 is required by Windows and Hyper-V guests to properly mitigate SMT related CPU 191 vulnerabilities. 192 193 When the option is set to 'auto' QEMU will enable the feature only when KVM 194 reports that non-architectural coresharing is impossible, this means that 195 hyper-threading is not supported or completely disabled on the host. This 196 setting also prevents migration as SMT settings on the destination may differ. 197 When the option is set to 'on' QEMU will always enable the feature, regardless 198 of host setup. To keep guests secure, this can only be used in conjunction with 199 exposing correct vCPU topology and vCPU pinning. 200 201``hv-version-id-build``, ``hv-version-id-major``, ``hv-version-id-minor``, ``hv-version-id-spack``, ``hv-version-id-sbranch``, ``hv-version-id-snumber`` 202 This changes Hyper-V version identification in CPUID 0x40000002.EAX-EDX from the 203 default (WS2016). 204 205 - ``hv-version-id-build`` sets 'Build Number' (32 bits) 206 - ``hv-version-id-major`` sets 'Major Version' (16 bits) 207 - ``hv-version-id-minor`` sets 'Minor Version' (16 bits) 208 - ``hv-version-id-spack`` sets 'Service Pack' (32 bits) 209 - ``hv-version-id-sbranch`` sets 'Service Branch' (8 bits) 210 - ``hv-version-id-snumber`` sets 'Service Number' (24 bits) 211 212 Note: hv-version-id-* are not enlightenments and thus don't enable Hyper-V 213 identification when specified without any other enlightenments. 214 215``hv-syndbg`` 216 Enables Hyper-V synthetic debugger interface, this is a special interface used 217 by Windows Kernel debugger to send the packets through, rather than sending 218 them via serial/network . 219 When enabled, this enlightenment provides additional communication facilities 220 to the guest: SynDbg messages. 221 This new communication is used by Windows Kernel debugger rather than sending 222 packets via serial/network, adding significant performance boost over the other 223 comm channels. 224 This enlightenment requires a VMBus device (-device vmbus-bridge,irq=15). 225 226 Requires: ``hv-relaxed``, ``hv_time``, ``hv-vapic``, ``hv-vpindex``, ``hv-synic``, ``hv-runtime``, ``hv-stimer`` 227 228``hv-emsr-bitmap`` 229 The enlightenment is nested specific, it targets Hyper-V on KVM guests. When 230 enabled, it allows L0 (KVM) and L1 (Hyper-V) hypervisors to collaborate to 231 avoid unnecessary updates to L2 MSR-Bitmap upon vmexits. While the protocol is 232 supported for both VMX (Intel) and SVM (AMD), the VMX implementation requires 233 Enlightened VMCS (``hv-evmcs``) feature to also be enabled. 234 235 Recommended: ``hv-evmcs`` (Intel) 236 237``hv-xmm-input`` 238 Hyper-V specification allows to pass parameters for certain hypercalls using XMM 239 registers ("XMM Fast Hypercall Input"). When the feature is in use, it allows 240 for faster hypercalls processing as KVM can avoid reading guest's memory. 241 242``hv-tlbflush-ext`` 243 Allow for extended GVA ranges to be passed to Hyper-V TLB flush hypercalls 244 (HvFlushVirtualAddressList/HvFlushVirtualAddressListEx). 245 246 Requires: ``hv-tlbflush`` 247 248``hv-tlbflush-direct`` 249 The enlightenment is nested specific, it targets Hyper-V on KVM guests. When 250 enabled, it allows L0 (KVM) to directly handle TLB flush hypercalls from L2 251 guest without the need to exit to L1 (Hyper-V) hypervisor. While the feature is 252 supported for both VMX (Intel) and SVM (AMD), the VMX implementation requires 253 Enlightened VMCS (``hv-evmcs``) feature to also be enabled. 254 255 Requires: ``hv-vapic`` 256 257 Recommended: ``hv-evmcs`` (Intel) 258 259Supplementary features 260---------------------- 261 262``hv-passthrough`` 263 In some cases (e.g. during development) it may make sense to use QEMU in 264 'pass-through' mode and give Windows guests all enlightenments currently 265 supported by KVM. 266 267 Note: ``hv-passthrough`` flag only enables enlightenments which are known to QEMU 268 (have corresponding 'hv-' flag) and copies ``hv-spinlocks`` and ``hv-vendor-id`` 269 values from KVM to QEMU. ``hv-passthrough`` overrides all other 'hv-' settings on 270 the command line. 271 272 Note: ``hv-passthrough`` does not enable ``hv-syndbg`` which can prevent certain 273 Windows guests from booting when used without proper configuration. If needed, 274 ``hv-syndbg`` can be enabled additionally. 275 276 Note: ``hv-passthrough`` effectively prevents migration as the list of enabled 277 enlightenments may differ between target and destination hosts. 278 279``hv-enforce-cpuid`` 280 By default, KVM allows the guest to use all currently supported Hyper-V 281 enlightenments when Hyper-V CPUID interface was exposed, regardless of if 282 some features were not announced in guest visible CPUIDs. ``hv-enforce-cpuid`` 283 feature alters this behavior and only allows the guest to use exposed Hyper-V 284 enlightenments. 285 286Recommendations 287--------------- 288 289To achieve the best performance of Windows and Hyper-V guests and unless there 290are any specific requirements (e.g. migration to older QEMU/KVM versions, 291emulating specific Hyper-V version, ...), it is recommended to enable all 292currently implemented Hyper-V enlightenments with the following exceptions: 293 294- ``hv-syndbg``, ``hv-passthrough``, ``hv-enforce-cpuid`` should not be enabled 295 in production configurations as these are debugging/development features. 296- ``hv-reset`` can be avoided as modern Hyper-V versions don't expose it. 297- ``hv-evmcs`` can (and should) be enabled on Intel CPUs only. While the feature 298 is only used in nested configurations (Hyper-V, WSL2), enabling it for regular 299 Windows guests should not have any negative effects. 300- ``hv-no-nonarch-coresharing`` must only be enabled if vCPUs are properly pinned 301 so no non-architectural core sharing is possible. 302- ``hv-vendor-id``, ``hv-version-id-build``, ``hv-version-id-major``, 303 ``hv-version-id-minor``, ``hv-version-id-spack``, ``hv-version-id-sbranch``, 304 ``hv-version-id-snumber`` can be left unchanged, guests are not supposed to 305 behave differently when different Hyper-V version is presented to them. 306- ``hv-crash`` must only be enabled if the crash information is consumed via 307 QAPI by higher levels of the virtualization stack. Enabling this feature 308 effectively prevents Windows from creating dumps upon crashes. 309- ``hv-reenlightenment`` can only be used on hardware which supports TSC 310 scaling or when guest migration is not needed. 311- ``hv-spinlocks`` should be set to e.g. 0xfff when host CPUs are overcommited 312 (meaning there are other scheduled tasks or guests) and can be left unchanged 313 from the default value (0xffffffff) otherwise. 314- ``hv-avic``/``hv-apicv`` should not be enabled if the hardware does not 315 support APIC virtualization (Intel APICv, AMD AVIC). 316 317Useful links 318------------ 319Hyper-V Top Level Functional specification and other information: 320 321- https://github.com/MicrosoftDocs/Virtualization-Documentation 322- https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/tlfs 323 324