1.. SPDX-License-Identifier: GPL-2.0 2 3====================== 4 USB4 and Thunderbolt 5====================== 6USB4 is the public specification based on Thunderbolt 3 protocol with 7some differences at the register level among other things. Connection 8manager is an entity running on the host router (host controller) 9responsible for enumerating routers and establishing tunnels. A 10connection manager can be implemented either in firmware or software. 11Typically PCs come with a firmware connection manager for Thunderbolt 3 12and early USB4 capable systems. Apple systems on the other hand use 13software connection manager and the later USB4 compliant devices follow 14the suit. 15 16The Linux Thunderbolt driver supports both and can detect at runtime which 17connection manager implementation is to be used. To be on the safe side the 18software connection manager in Linux also advertises security level 19``user`` which means PCIe tunneling is disabled by default. The 20documentation below applies to both implementations with the exception that 21the software connection manager only supports ``user`` security level and 22is expected to be accompanied with an IOMMU based DMA protection. 23 24Security levels and how to use them 25----------------------------------- 26The interface presented here is not meant for end users. Instead there 27should be a userspace tool that handles all the low-level details, keeps 28a database of the authorized devices and prompts users for new connections. 29 30More details about the sysfs interface for Thunderbolt devices can be 31found in ``Documentation/ABI/testing/sysfs-bus-thunderbolt``. 32 33Those users who just want to connect any device without any sort of 34manual work can add following line to 35``/etc/udev/rules.d/99-local.rules``:: 36 37 ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1" 38 39This will authorize all devices automatically when they appear. However, 40keep in mind that this bypasses the security levels and makes the system 41vulnerable to DMA attacks. 42 43Starting with Intel Falcon Ridge Thunderbolt controller there are 4 44security levels available. Intel Titan Ridge added one more security level 45(usbonly). The reason for these is the fact that the connected devices can 46be DMA masters and thus read contents of the host memory without CPU and OS 47knowing about it. There are ways to prevent this by setting up an IOMMU but 48it is not always available for various reasons. 49 50The security levels are as follows: 51 52 none 53 All devices are automatically connected by the firmware. No user 54 approval is needed. In BIOS settings this is typically called 55 *Legacy mode*. 56 57 user 58 User is asked whether the device is allowed to be connected. 59 Based on the device identification information available through 60 ``/sys/bus/thunderbolt/devices``, the user then can make the decision. 61 In BIOS settings this is typically called *Unique ID*. 62 63 secure 64 User is asked whether the device is allowed to be connected. In 65 addition to UUID the device (if it supports secure connect) is sent 66 a challenge that should match the expected one based on a random key 67 written to the ``key`` sysfs attribute. In BIOS settings this is 68 typically called *One time saved key*. 69 70 dponly 71 The firmware automatically creates tunnels for Display Port and 72 USB. No PCIe tunneling is done. In BIOS settings this is 73 typically called *Display Port Only*. 74 75 usbonly 76 The firmware automatically creates tunnels for the USB controller and 77 Display Port in a dock. All PCIe links downstream of the dock are 78 removed. 79 80The current security level can be read from 81``/sys/bus/thunderbolt/devices/domainX/security`` where ``domainX`` is 82the Thunderbolt domain the host controller manages. There is typically 83one domain per Thunderbolt host controller. 84 85If the security level reads as ``user`` or ``secure`` the connected 86device must be authorized by the user before PCIe tunnels are created 87(e.g the PCIe device appears). 88 89Each Thunderbolt device plugged in will appear in sysfs under 90``/sys/bus/thunderbolt/devices``. The device directory carries 91information that can be used to identify the particular device, 92including its name and UUID. 93 94Authorizing devices when security level is ``user`` or ``secure`` 95----------------------------------------------------------------- 96When a device is plugged in it will appear in sysfs as follows:: 97 98 /sys/bus/thunderbolt/devices/0-1/authorized - 0 99 /sys/bus/thunderbolt/devices/0-1/device - 0x8004 100 /sys/bus/thunderbolt/devices/0-1/device_name - Thunderbolt to FireWire Adapter 101 /sys/bus/thunderbolt/devices/0-1/vendor - 0x1 102 /sys/bus/thunderbolt/devices/0-1/vendor_name - Apple, Inc. 103 /sys/bus/thunderbolt/devices/0-1/unique_id - e0376f00-0300-0100-ffff-ffffffffffff 104 105The ``authorized`` attribute reads 0 which means no PCIe tunnels are 106created yet. The user can authorize the device by simply entering:: 107 108 # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized 109 110This will create the PCIe tunnels and the device is now connected. 111 112If the device supports secure connect, and the domain security level is 113set to ``secure``, it has an additional attribute ``key`` which can hold 114a random 32-byte value used for authorization and challenging the device in 115future connects:: 116 117 /sys/bus/thunderbolt/devices/0-3/authorized - 0 118 /sys/bus/thunderbolt/devices/0-3/device - 0x305 119 /sys/bus/thunderbolt/devices/0-3/device_name - AKiTiO Thunder3 PCIe Box 120 /sys/bus/thunderbolt/devices/0-3/key - 121 /sys/bus/thunderbolt/devices/0-3/vendor - 0x41 122 /sys/bus/thunderbolt/devices/0-3/vendor_name - inXtron 123 /sys/bus/thunderbolt/devices/0-3/unique_id - dc010000-0000-8508-a22d-32ca6421cb16 124 125Notice the key is empty by default. 126 127If the user does not want to use secure connect they can just ``echo 1`` 128to the ``authorized`` attribute and the PCIe tunnels will be created in 129the same way as in the ``user`` security level. 130 131If the user wants to use secure connect, the first time the device is 132plugged a key needs to be created and sent to the device:: 133 134 # key=$(openssl rand -hex 32) 135 # echo $key > /sys/bus/thunderbolt/devices/0-3/key 136 # echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized 137 138Now the device is connected (PCIe tunnels are created) and in addition 139the key is stored on the device NVM. 140 141Next time the device is plugged in the user can verify (challenge) the 142device using the same key:: 143 144 # echo $key > /sys/bus/thunderbolt/devices/0-3/key 145 # echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized 146 147If the challenge the device returns back matches the one we expect based 148on the key, the device is connected and the PCIe tunnels are created. 149However, if the challenge fails no tunnels are created and error is 150returned to the user. 151 152If the user still wants to connect the device they can either approve 153the device without a key or write a new key and write 1 to the 154``authorized`` file to get the new key stored on the device NVM. 155 156DMA protection utilizing IOMMU 157------------------------------ 158Recent systems from 2018 and forward with Thunderbolt ports may natively 159support IOMMU. This means that Thunderbolt security is handled by an IOMMU 160so connected devices cannot access memory regions outside of what is 161allocated for them by drivers. When Linux is running on such system it 162automatically enables IOMMU if not enabled by the user already. These 163systems can be identified by reading ``1`` from 164``/sys/bus/thunderbolt/devices/domainX/iommu_dma_protection`` attribute. 165 166The driver does not do anything special in this case but because DMA 167protection is handled by the IOMMU, security levels (if set) are 168redundant. For this reason some systems ship with security level set to 169``none``. Other systems have security level set to ``user`` in order to 170support downgrade to older OS, so users who want to automatically 171authorize devices when IOMMU DMA protection is enabled can use the 172following ``udev`` rule:: 173 174 ACTION=="add", SUBSYSTEM=="thunderbolt", ATTRS{iommu_dma_protection}=="1", ATTR{authorized}=="0", ATTR{authorized}="1" 175 176Upgrading NVM on Thunderbolt device or host 177------------------------------------------- 178Since most of the functionality is handled in firmware running on a 179host controller or a device, it is important that the firmware can be 180upgraded to the latest where possible bugs in it have been fixed. 181Typically OEMs provide this firmware from their support site. 182 183There is also a central site which has links where to download firmware 184for some machines: 185 186 `Thunderbolt Updates <https://thunderbolttechnology.net/updates>`_ 187 188Before you upgrade firmware on a device or host, please make sure it is a 189suitable upgrade. Failing to do that may render the device (or host) in a 190state where it cannot be used properly anymore without special tools! 191 192Host NVM upgrade on Apple Macs is not supported. 193 194Once the NVM image has been downloaded, you need to plug in a 195Thunderbolt device so that the host controller appears. It does not 196matter which device is connected (unless you are upgrading NVM on a 197device - then you need to connect that particular device). 198 199Note an OEM-specific method to power the controller up ("force power") may 200be available for your system in which case there is no need to plug in a 201Thunderbolt device. 202 203After that we can write the firmware to the non-active parts of the NVM 204of the host or device. As an example here is how Intel NUC6i7KYK (Skull 205Canyon) Thunderbolt controller NVM is upgraded:: 206 207 # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem 208 209Once the operation completes we can trigger NVM authentication and 210upgrade process as follows:: 211 212 # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate 213 214If no errors are returned, the host controller shortly disappears. Once 215it comes back the driver notices it and initiates a full power cycle. 216After a while the host controller appears again and this time it should 217be fully functional. 218 219We can verify that the new NVM firmware is active by running the following 220commands:: 221 222 # cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate 223 0x0 224 # cat /sys/bus/thunderbolt/devices/0-0/nvm_version 225 18.0 226 227If ``nvm_authenticate`` contains anything other than 0x0 it is the error 228code from the last authentication cycle, which means the authentication 229of the NVM image failed. 230 231Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN`` 232depend on the order they are registered in the NVMem subsystem. N in 233the name is the identifier added by the NVMem subsystem. 234 235Upgrading NVM when host controller is in safe mode 236-------------------------------------------------- 237If the existing NVM is not properly authenticated (or is missing) the 238host controller goes into safe mode which means that the only available 239functionality is flashing a new NVM image. When in this mode, reading 240``nvm_version`` fails with ``ENODATA`` and the device identification 241information is missing. 242 243To recover from this mode, one needs to flash a valid NVM image to the 244host controller in the same way it is done in the previous chapter. 245 246Networking over Thunderbolt cable 247--------------------------------- 248Thunderbolt technology allows software communication between two hosts 249connected by a Thunderbolt cable. 250 251It is possible to tunnel any kind of traffic over a Thunderbolt link but 252currently we only support Apple ThunderboltIP protocol. 253 254If the other host is running Windows or macOS, the only thing you need to 255do is to connect a Thunderbolt cable between the two hosts; the 256``thunderbolt-net`` driver is loaded automatically. If the other host is 257also Linux you should load ``thunderbolt-net`` manually on one host (it 258does not matter which one):: 259 260 # modprobe thunderbolt-net 261 262This triggers module load on the other host automatically. If the driver 263is built-in to the kernel image, there is no need to do anything. 264 265The driver will create one virtual ethernet interface per Thunderbolt 266port which are named like ``thunderbolt0`` and so on. From this point 267you can either use standard userspace tools like ``ifconfig`` to 268configure the interface or let your GUI handle it automatically. 269 270Forcing power 271------------- 272Many OEMs include a method that can be used to force the power of a 273Thunderbolt controller to an "On" state even if nothing is connected. 274If supported by your machine this will be exposed by the WMI bus with 275a sysfs attribute called "force_power". 276 277For example the intel-wmi-thunderbolt driver exposes this attribute in: 278 /sys/bus/wmi/devices/86CCFD48-205E-4A77-9C48-2021CBEDE341/force_power 279 280 To force the power to on, write 1 to this attribute file. 281 To disable force power, write 0 to this attribute file. 282 283Note: it's currently not possible to query the force power state of a platform. 284