1.. SPDX-License-Identifier: GPL-2.0 2 3====================== 4 USB4 and Thunderbolt 5====================== 6USB4 is the public specification based on Thunderbolt 3 protocol with 7some differences at the register level among other things. Connection 8manager is an entity running on the host router (host controller) 9responsible for enumerating routers and establishing tunnels. A 10connection manager can be implemented either in firmware or software. 11Typically PCs come with a firmware connection manager for Thunderbolt 3 12and early USB4 capable systems. Apple systems on the other hand use 13software connection manager and the later USB4 compliant devices follow 14the suit. 15 16The Linux Thunderbolt driver supports both and can detect at runtime which 17connection manager implementation is to be used. To be on the safe side the 18software connection manager in Linux also advertises security level 19``user`` which means PCIe tunneling is disabled by default. The 20documentation below applies to both implementations with the exception that 21the software connection manager only supports ``user`` security level and 22is expected to be accompanied with an IOMMU based DMA protection. 23 24Security levels and how to use them 25----------------------------------- 26The interface presented here is not meant for end users. Instead there 27should be a userspace tool that handles all the low-level details, keeps 28a database of the authorized devices and prompts users for new connections. 29 30More details about the sysfs interface for Thunderbolt devices can be 31found in ``Documentation/ABI/testing/sysfs-bus-thunderbolt``. 32 33Those users who just want to connect any device without any sort of 34manual work can add following line to 35``/etc/udev/rules.d/99-local.rules``:: 36 37 ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1" 38 39This will authorize all devices automatically when they appear. However, 40keep in mind that this bypasses the security levels and makes the system 41vulnerable to DMA attacks. 42 43Starting with Intel Falcon Ridge Thunderbolt controller there are 4 44security levels available. Intel Titan Ridge added one more security level 45(usbonly). The reason for these is the fact that the connected devices can 46be DMA masters and thus read contents of the host memory without CPU and OS 47knowing about it. There are ways to prevent this by setting up an IOMMU but 48it is not always available for various reasons. 49 50The security levels are as follows: 51 52 none 53 All devices are automatically connected by the firmware. No user 54 approval is needed. In BIOS settings this is typically called 55 *Legacy mode*. 56 57 user 58 User is asked whether the device is allowed to be connected. 59 Based on the device identification information available through 60 ``/sys/bus/thunderbolt/devices``, the user then can make the decision. 61 In BIOS settings this is typically called *Unique ID*. 62 63 secure 64 User is asked whether the device is allowed to be connected. In 65 addition to UUID the device (if it supports secure connect) is sent 66 a challenge that should match the expected one based on a random key 67 written to the ``key`` sysfs attribute. In BIOS settings this is 68 typically called *One time saved key*. 69 70 dponly 71 The firmware automatically creates tunnels for Display Port and 72 USB. No PCIe tunneling is done. In BIOS settings this is 73 typically called *Display Port Only*. 74 75 usbonly 76 The firmware automatically creates tunnels for the USB controller and 77 Display Port in a dock. All PCIe links downstream of the dock are 78 removed. 79 80The current security level can be read from 81``/sys/bus/thunderbolt/devices/domainX/security`` where ``domainX`` is 82the Thunderbolt domain the host controller manages. There is typically 83one domain per Thunderbolt host controller. 84 85If the security level reads as ``user`` or ``secure`` the connected 86device must be authorized by the user before PCIe tunnels are created 87(e.g the PCIe device appears). 88 89Each Thunderbolt device plugged in will appear in sysfs under 90``/sys/bus/thunderbolt/devices``. The device directory carries 91information that can be used to identify the particular device, 92including its name and UUID. 93 94Authorizing devices when security level is ``user`` or ``secure`` 95----------------------------------------------------------------- 96When a device is plugged in it will appear in sysfs as follows:: 97 98 /sys/bus/thunderbolt/devices/0-1/authorized - 0 99 /sys/bus/thunderbolt/devices/0-1/device - 0x8004 100 /sys/bus/thunderbolt/devices/0-1/device_name - Thunderbolt to FireWire Adapter 101 /sys/bus/thunderbolt/devices/0-1/vendor - 0x1 102 /sys/bus/thunderbolt/devices/0-1/vendor_name - Apple, Inc. 103 /sys/bus/thunderbolt/devices/0-1/unique_id - e0376f00-0300-0100-ffff-ffffffffffff 104 105The ``authorized`` attribute reads 0 which means no PCIe tunnels are 106created yet. The user can authorize the device by simply entering:: 107 108 # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized 109 110This will create the PCIe tunnels and the device is now connected. 111 112If the device supports secure connect, and the domain security level is 113set to ``secure``, it has an additional attribute ``key`` which can hold 114a random 32-byte value used for authorization and challenging the device in 115future connects:: 116 117 /sys/bus/thunderbolt/devices/0-3/authorized - 0 118 /sys/bus/thunderbolt/devices/0-3/device - 0x305 119 /sys/bus/thunderbolt/devices/0-3/device_name - AKiTiO Thunder3 PCIe Box 120 /sys/bus/thunderbolt/devices/0-3/key - 121 /sys/bus/thunderbolt/devices/0-3/vendor - 0x41 122 /sys/bus/thunderbolt/devices/0-3/vendor_name - inXtron 123 /sys/bus/thunderbolt/devices/0-3/unique_id - dc010000-0000-8508-a22d-32ca6421cb16 124 125Notice the key is empty by default. 126 127If the user does not want to use secure connect they can just ``echo 1`` 128to the ``authorized`` attribute and the PCIe tunnels will be created in 129the same way as in the ``user`` security level. 130 131If the user wants to use secure connect, the first time the device is 132plugged a key needs to be created and sent to the device:: 133 134 # key=$(openssl rand -hex 32) 135 # echo $key > /sys/bus/thunderbolt/devices/0-3/key 136 # echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized 137 138Now the device is connected (PCIe tunnels are created) and in addition 139the key is stored on the device NVM. 140 141Next time the device is plugged in the user can verify (challenge) the 142device using the same key:: 143 144 # echo $key > /sys/bus/thunderbolt/devices/0-3/key 145 # echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized 146 147If the challenge the device returns back matches the one we expect based 148on the key, the device is connected and the PCIe tunnels are created. 149However, if the challenge fails no tunnels are created and error is 150returned to the user. 151 152If the user still wants to connect the device they can either approve 153the device without a key or write a new key and write 1 to the 154``authorized`` file to get the new key stored on the device NVM. 155 156DMA protection utilizing IOMMU 157------------------------------ 158Recent systems from 2018 and forward with Thunderbolt ports may natively 159support IOMMU. This means that Thunderbolt security is handled by an IOMMU 160so connected devices cannot access memory regions outside of what is 161allocated for them by drivers. When Linux is running on such system it 162automatically enables IOMMU if not enabled by the user already. These 163systems can be identified by reading ``1`` from 164``/sys/bus/thunderbolt/devices/domainX/iommu_dma_protection`` attribute. 165 166The driver does not do anything special in this case but because DMA 167protection is handled by the IOMMU, security levels (if set) are 168redundant. For this reason some systems ship with security level set to 169``none``. Other systems have security level set to ``user`` in order to 170support downgrade to older OS, so users who want to automatically 171authorize devices when IOMMU DMA protection is enabled can use the 172following ``udev`` rule:: 173 174 ACTION=="add", SUBSYSTEM=="thunderbolt", ATTRS{iommu_dma_protection}=="1", ATTR{authorized}=="0", ATTR{authorized}="1" 175 176Upgrading NVM on Thunderbolt device, host or retimer 177---------------------------------------------------- 178Since most of the functionality is handled in firmware running on a 179host controller or a device, it is important that the firmware can be 180upgraded to the latest where possible bugs in it have been fixed. 181Typically OEMs provide this firmware from their support site. 182 183There is also a central site which has links where to download firmware 184for some machines: 185 186 `Thunderbolt Updates <https://thunderbolttechnology.net/updates>`_ 187 188Before you upgrade firmware on a device, host or retimer, please make 189sure it is a suitable upgrade. Failing to do that may render the device 190in a state where it cannot be used properly anymore without special 191tools! 192 193Host NVM upgrade on Apple Macs is not supported. 194 195Once the NVM image has been downloaded, you need to plug in a 196Thunderbolt device so that the host controller appears. It does not 197matter which device is connected (unless you are upgrading NVM on a 198device - then you need to connect that particular device). 199 200Note an OEM-specific method to power the controller up ("force power") may 201be available for your system in which case there is no need to plug in a 202Thunderbolt device. 203 204After that we can write the firmware to the non-active parts of the NVM 205of the host or device. As an example here is how Intel NUC6i7KYK (Skull 206Canyon) Thunderbolt controller NVM is upgraded:: 207 208 # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem 209 210Once the operation completes we can trigger NVM authentication and 211upgrade process as follows:: 212 213 # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate 214 215If no errors are returned, the host controller shortly disappears. Once 216it comes back the driver notices it and initiates a full power cycle. 217After a while the host controller appears again and this time it should 218be fully functional. 219 220We can verify that the new NVM firmware is active by running the following 221commands:: 222 223 # cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate 224 0x0 225 # cat /sys/bus/thunderbolt/devices/0-0/nvm_version 226 18.0 227 228If ``nvm_authenticate`` contains anything other than 0x0 it is the error 229code from the last authentication cycle, which means the authentication 230of the NVM image failed. 231 232Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN`` 233depend on the order they are registered in the NVMem subsystem. N in 234the name is the identifier added by the NVMem subsystem. 235 236Upgrading NVM when host controller is in safe mode 237-------------------------------------------------- 238If the existing NVM is not properly authenticated (or is missing) the 239host controller goes into safe mode which means that the only available 240functionality is flashing a new NVM image. When in this mode, reading 241``nvm_version`` fails with ``ENODATA`` and the device identification 242information is missing. 243 244To recover from this mode, one needs to flash a valid NVM image to the 245host controller in the same way it is done in the previous chapter. 246 247Networking over Thunderbolt cable 248--------------------------------- 249Thunderbolt technology allows software communication between two hosts 250connected by a Thunderbolt cable. 251 252It is possible to tunnel any kind of traffic over a Thunderbolt link but 253currently we only support Apple ThunderboltIP protocol. 254 255If the other host is running Windows or macOS, the only thing you need to 256do is to connect a Thunderbolt cable between the two hosts; the 257``thunderbolt-net`` driver is loaded automatically. If the other host is 258also Linux you should load ``thunderbolt-net`` manually on one host (it 259does not matter which one):: 260 261 # modprobe thunderbolt-net 262 263This triggers module load on the other host automatically. If the driver 264is built-in to the kernel image, there is no need to do anything. 265 266The driver will create one virtual ethernet interface per Thunderbolt 267port which are named like ``thunderbolt0`` and so on. From this point 268you can either use standard userspace tools like ``ifconfig`` to 269configure the interface or let your GUI handle it automatically. 270 271Forcing power 272------------- 273Many OEMs include a method that can be used to force the power of a 274Thunderbolt controller to an "On" state even if nothing is connected. 275If supported by your machine this will be exposed by the WMI bus with 276a sysfs attribute called "force_power". 277 278For example the intel-wmi-thunderbolt driver exposes this attribute in: 279 /sys/bus/wmi/devices/86CCFD48-205E-4A77-9C48-2021CBEDE341/force_power 280 281 To force the power to on, write 1 to this attribute file. 282 To disable force power, write 0 to this attribute file. 283 284Note: it's currently not possible to query the force power state of a platform. 285