1Software Guard eXtensions (SGX) 2=============================== 3 4Overview 5-------- 6 7Intel Software Guard eXtensions (SGX) is a set of instructions and mechanisms 8for memory accesses in order to provide security accesses for sensitive 9applications and data. SGX allows an application to use its particular 10address space as an *enclave*, which is a protected area provides confidentiality 11and integrity even in the presence of privileged malware. Accesses to the 12enclave memory area from any software not resident in the enclave are prevented, 13including those from privileged software. 14 15Virtual SGX 16----------- 17 18SGX feature is exposed to guest via SGX CPUID. Looking at SGX CPUID, we can 19report the same CPUID info to guest as on host for most of SGX CPUID. With 20reporting the same CPUID guest is able to use full capacity of SGX, and KVM 21doesn't need to emulate those info. 22 23The guest's EPC base and size are determined by QEMU, and KVM needs QEMU to 24notify such info to it before it can initialize SGX for guest. 25 26Virtual EPC 27~~~~~~~~~~~ 28 29By default, QEMU does not assign EPC to a VM, i.e. fully enabling SGX in a VM 30requires explicit allocation of EPC to the VM. Similar to other specialized 31memory types, e.g. hugetlbfs, EPC is exposed as a memory backend. 32 33SGX EPC is enumerated through CPUID, i.e. EPC "devices" need to be realized 34prior to realizing the vCPUs themselves, which occurs long before generic 35devices are parsed and realized. This limitation means that EPC does not 36require -maxmem as EPC is not treated as {cold,hot}plugged memory. 37 38QEMU does not artificially restrict the number of EPC sections exposed to a 39guest, e.g. QEMU will happily allow you to create 64 1M EPC sections. Be aware 40that some kernels may not recognize all EPC sections, e.g. the Linux SGX driver 41is hardwired to support only 8 EPC sections. 42 43The following QEMU snippet creates two EPC sections, with 64M pre-allocated 44to the VM and an additional 28M mapped but not allocated:: 45 46 -object memory-backend-epc,id=mem1,size=64M,prealloc=on \ 47 -object memory-backend-epc,id=mem2,size=28M \ 48 -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2 49 50Note: 51 52The size and location of the virtual EPC are far less restricted compared 53to physical EPC. Because physical EPC is protected via range registers, 54the size of the physical EPC must be a power of two (though software sees 55a subset of the full EPC, e.g. 92M or 128M) and the EPC must be naturally 56aligned. KVM SGX's virtual EPC is purely a software construct and only 57requires the size and location to be page aligned. QEMU enforces the EPC 58size is a multiple of 4k and will ensure the base of the EPC is 4k aligned. 59To simplify the implementation, EPC is always located above 4g in the guest 60physical address space. 61 62Migration 63~~~~~~~~~ 64 65QEMU/KVM doesn't prevent live migrating SGX VMs, although from hardware's 66perspective, SGX doesn't support live migration, since both EPC and the SGX 67key hierarchy are bound to the physical platform. However live migration 68can be supported in the sense if guest software stack can support recreating 69enclaves when it suffers sudden lose of EPC; and if guest enclaves can detect 70SGX keys being changed, and handle gracefully. For instance, when ERESUME fails 71with #PF.SGX, guest software can gracefully detect it and recreate enclaves; 72and when enclave fails to unseal sensitive information from outside, it can 73detect such error and sensitive information can be provisioned to it again. 74 75CPUID 76~~~~~ 77 78Due to its myriad dependencies, SGX is currently not listed as supported 79in any of QEMU's built-in CPU configuration. To expose SGX (and SGX Launch 80Control) to a guest, you must either use ``-cpu host`` to pass-through the 81host CPU model, or explicitly enable SGX when using a built-in CPU model, 82e.g. via ``-cpu <model>,+sgx`` or ``-cpu <model>,+sgx,+sgxlc``. 83 84All SGX sub-features enumerated through CPUID, e.g. SGX2, MISCSELECT, 85ATTRIBUTES, etc... can be restricted via CPUID flags. Be aware that enforcing 86restriction of MISCSELECT, ATTRIBUTES and XFRM requires intercepting ECREATE, 87i.e. may marginally reduce SGX performance in the guest. All SGX sub-features 88controlled via -cpu are prefixed with "sgx", e.g.:: 89 90 $ qemu-system-x86_64 -cpu help | xargs printf "%s\n" | grep sgx 91 sgx 92 sgx-debug 93 sgx-encls-c 94 sgx-enclv 95 sgx-exinfo 96 sgx-kss 97 sgx-mode64 98 sgx-provisionkey 99 sgx-tokenkey 100 sgx1 101 sgx2 102 sgxlc 103 104The following QEMU snippet passes through the host CPU but restricts access to 105the provision and EINIT token keys:: 106 107 -cpu host,-sgx-provisionkey,-sgx-tokenkey 108 109SGX sub-features cannot be emulated, i.e. sub-features that are not present 110in hardware cannot be forced on via '-cpu'. 111 112Virtualize SGX Launch Control 113~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 114 115QEMU SGX support for Launch Control (LC) is passive, in the sense that it 116does not actively change the LC configuration. QEMU SGX provides the user 117the ability to set/clear the CPUID flag (and by extension the associated 118IA32_FEATURE_CONTROL MSR bit in fw_cfg) and saves/restores the LE Hash MSRs 119when getting/putting guest state, but QEMU does not add new controls to 120directly modify the LC configuration. Similar to hardware behavior, locking 121the LC configuration to a non-Intel value is left to guest firmware. Unlike 122host bios setting for SGX launch control(LC), there is no special bios setting 123for SGX guest by our design. If host is in locked mode, we can still allow 124creating VM with SGX. 125 126Feature Control 127~~~~~~~~~~~~~~~ 128 129QEMU SGX updates the ``etc/msr_feature_control`` fw_cfg entry to set the SGX 130(bit 18) and SGX LC (bit 17) flags based on their respective CPUID support, 131i.e. existing guest firmware will automatically set SGX and SGX LC accordingly, 132assuming said firmware supports fw_cfg.msr_feature_control. 133 134Launching a guest 135----------------- 136 137To launch a SGX guest: 138 139.. parsed-literal:: 140 141 |qemu_system_x86| \\ 142 -cpu host,+sgx-provisionkey \\ 143 -object memory-backend-epc,id=mem1,size=64M,prealloc=on \\ 144 -M sgx-epc.0.memdev=mem1,sgx-epc.0.node=0 145 146Utilizing SGX in the guest requires a kernel/OS with SGX support. 147The support can be determined in guest by:: 148 149 $ grep sgx /proc/cpuinfo 150 151and SGX epc info by:: 152 153 $ dmesg | grep sgx 154 [ 0.182807] sgx: EPC section 0x140000000-0x143ffffff 155 [ 0.183695] sgx: [Firmware Bug]: Unable to map EPC section to online node. Fallback to the NUMA node 0. 156 157To launch a SGX numa guest: 158 159.. parsed-literal:: 160 161 |qemu_system_x86| \\ 162 -cpu host,+sgx-provisionkey \\ 163 -object memory-backend-ram,size=2G,host-nodes=0,policy=bind,id=node0 \\ 164 -object memory-backend-epc,id=mem0,size=64M,prealloc=on,host-nodes=0,policy=bind \\ 165 -numa node,nodeid=0,cpus=0-1,memdev=node0 \\ 166 -object memory-backend-ram,size=2G,host-nodes=1,policy=bind,id=node1 \\ 167 -object memory-backend-epc,id=mem1,size=28M,prealloc=on,host-nodes=1,policy=bind \\ 168 -numa node,nodeid=1,cpus=2-3,memdev=node1 \\ 169 -M sgx-epc.0.memdev=mem0,sgx-epc.0.node=0,sgx-epc.1.memdev=mem1,sgx-epc.1.node=1 170 171and SGX epc numa info by:: 172 173 $ dmesg | grep sgx 174 [ 0.369937] sgx: EPC section 0x180000000-0x183ffffff 175 [ 0.370259] sgx: EPC section 0x184000000-0x185bfffff 176 177 $ dmesg | grep SRAT 178 [ 0.009981] ACPI: SRAT: Node 0 PXM 0 [mem 0x180000000-0x183ffffff] 179 [ 0.009982] ACPI: SRAT: Node 1 PXM 1 [mem 0x184000000-0x185bfffff] 180 181References 182---------- 183 184- `SGX Homepage <https://software.intel.com/sgx>`__ 185 186- `SGX SDK <https://github.com/intel/linux-sgx.git>`__ 187 188- SGX specification: Intel SDM Volume 3 189