1.. SPDX-License-Identifier: GPL-2.0
2
3======================
4 USB4 and Thunderbolt
5======================
6USB4 is the public specification based on Thunderbolt 3 protocol with
7some differences at the register level among other things. Connection
8manager is an entity running on the host router (host controller)
9responsible for enumerating routers and establishing tunnels. A
10connection manager can be implemented either in firmware or software.
11Typically PCs come with a firmware connection manager for Thunderbolt 3
12and early USB4 capable systems. Apple systems on the other hand use
13software connection manager and the later USB4 compliant devices follow
14the suit.
15
16The Linux Thunderbolt driver supports both and can detect at runtime which
17connection manager implementation is to be used. To be on the safe side the
18software connection manager in Linux also advertises security level
19``user`` which means PCIe tunneling is disabled by default. The
20documentation below applies to both implementations with the exception that
21the software connection manager only supports ``user`` security level and
22is expected to be accompanied with an IOMMU based DMA protection.
23
24Security levels and how to use them
25-----------------------------------
26The interface presented here is not meant for end users. Instead there
27should be a userspace tool that handles all the low-level details, keeps
28a database of the authorized devices and prompts users for new connections.
29
30More details about the sysfs interface for Thunderbolt devices can be
31found in ``Documentation/ABI/testing/sysfs-bus-thunderbolt``.
32
33Those users who just want to connect any device without any sort of
34manual work can add following line to
35``/etc/udev/rules.d/99-local.rules``::
36
37  ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1"
38
39This will authorize all devices automatically when they appear. However,
40keep in mind that this bypasses the security levels and makes the system
41vulnerable to DMA attacks.
42
43Starting with Intel Falcon Ridge Thunderbolt controller there are 4
44security levels available. Intel Titan Ridge added one more security level
45(usbonly). The reason for these is the fact that the connected devices can
46be DMA masters and thus read contents of the host memory without CPU and OS
47knowing about it. There are ways to prevent this by setting up an IOMMU but
48it is not always available for various reasons.
49
50Some USB4 systems have a BIOS setting to disable PCIe tunneling. This is
51treated as another security level (nopcie).
52
53The security levels are as follows:
54
55  none
56    All devices are automatically connected by the firmware. No user
57    approval is needed. In BIOS settings this is typically called
58    *Legacy mode*.
59
60  user
61    User is asked whether the device is allowed to be connected.
62    Based on the device identification information available through
63    ``/sys/bus/thunderbolt/devices``, the user then can make the decision.
64    In BIOS settings this is typically called *Unique ID*.
65
66  secure
67    User is asked whether the device is allowed to be connected. In
68    addition to UUID the device (if it supports secure connect) is sent
69    a challenge that should match the expected one based on a random key
70    written to the ``key`` sysfs attribute. In BIOS settings this is
71    typically called *One time saved key*.
72
73  dponly
74    The firmware automatically creates tunnels for Display Port and
75    USB. No PCIe tunneling is done. In BIOS settings this is
76    typically called *Display Port Only*.
77
78  usbonly
79    The firmware automatically creates tunnels for the USB controller and
80    Display Port in a dock. All PCIe links downstream of the dock are
81    removed.
82
83  nopcie
84    PCIe tunneling is disabled/forbidden from the BIOS. Available in some
85    USB4 systems.
86
87The current security level can be read from
88``/sys/bus/thunderbolt/devices/domainX/security`` where ``domainX`` is
89the Thunderbolt domain the host controller manages. There is typically
90one domain per Thunderbolt host controller.
91
92If the security level reads as ``user`` or ``secure`` the connected
93device must be authorized by the user before PCIe tunnels are created
94(e.g the PCIe device appears).
95
96Each Thunderbolt device plugged in will appear in sysfs under
97``/sys/bus/thunderbolt/devices``. The device directory carries
98information that can be used to identify the particular device,
99including its name and UUID.
100
101Authorizing devices when security level is ``user`` or ``secure``
102-----------------------------------------------------------------
103When a device is plugged in it will appear in sysfs as follows::
104
105  /sys/bus/thunderbolt/devices/0-1/authorized	- 0
106  /sys/bus/thunderbolt/devices/0-1/device	- 0x8004
107  /sys/bus/thunderbolt/devices/0-1/device_name	- Thunderbolt to FireWire Adapter
108  /sys/bus/thunderbolt/devices/0-1/vendor	- 0x1
109  /sys/bus/thunderbolt/devices/0-1/vendor_name	- Apple, Inc.
110  /sys/bus/thunderbolt/devices/0-1/unique_id	- e0376f00-0300-0100-ffff-ffffffffffff
111
112The ``authorized`` attribute reads 0 which means no PCIe tunnels are
113created yet. The user can authorize the device by simply entering::
114
115  # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized
116
117This will create the PCIe tunnels and the device is now connected.
118
119If the device supports secure connect, and the domain security level is
120set to ``secure``, it has an additional attribute ``key`` which can hold
121a random 32-byte value used for authorization and challenging the device in
122future connects::
123
124  /sys/bus/thunderbolt/devices/0-3/authorized	- 0
125  /sys/bus/thunderbolt/devices/0-3/device	- 0x305
126  /sys/bus/thunderbolt/devices/0-3/device_name	- AKiTiO Thunder3 PCIe Box
127  /sys/bus/thunderbolt/devices/0-3/key		-
128  /sys/bus/thunderbolt/devices/0-3/vendor	- 0x41
129  /sys/bus/thunderbolt/devices/0-3/vendor_name	- inXtron
130  /sys/bus/thunderbolt/devices/0-3/unique_id	- dc010000-0000-8508-a22d-32ca6421cb16
131
132Notice the key is empty by default.
133
134If the user does not want to use secure connect they can just ``echo 1``
135to the ``authorized`` attribute and the PCIe tunnels will be created in
136the same way as in the ``user`` security level.
137
138If the user wants to use secure connect, the first time the device is
139plugged a key needs to be created and sent to the device::
140
141  # key=$(openssl rand -hex 32)
142  # echo $key > /sys/bus/thunderbolt/devices/0-3/key
143  # echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized
144
145Now the device is connected (PCIe tunnels are created) and in addition
146the key is stored on the device NVM.
147
148Next time the device is plugged in the user can verify (challenge) the
149device using the same key::
150
151  # echo $key > /sys/bus/thunderbolt/devices/0-3/key
152  # echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized
153
154If the challenge the device returns back matches the one we expect based
155on the key, the device is connected and the PCIe tunnels are created.
156However, if the challenge fails no tunnels are created and error is
157returned to the user.
158
159If the user still wants to connect the device they can either approve
160the device without a key or write a new key and write 1 to the
161``authorized`` file to get the new key stored on the device NVM.
162
163De-authorizing devices
164----------------------
165It is possible to de-authorize devices by writing ``0`` to their
166``authorized`` attribute. This requires support from the connection
167manager implementation and can be checked by reading domain
168``deauthorization`` attribute. If it reads ``1`` then the feature is
169supported.
170
171When a device is de-authorized the PCIe tunnel from the parent device
172PCIe downstream (or root) port to the device PCIe upstream port is torn
173down. This is essentially the same thing as PCIe hot-remove and the PCIe
174toplogy in question will not be accessible anymore until the device is
175authorized again. If there is storage such as NVMe or similar involved,
176there is a risk for data loss if the filesystem on that storage is not
177properly shut down. You have been warned!
178
179DMA protection utilizing IOMMU
180------------------------------
181Recent systems from 2018 and forward with Thunderbolt ports may natively
182support IOMMU. This means that Thunderbolt security is handled by an IOMMU
183so connected devices cannot access memory regions outside of what is
184allocated for them by drivers. When Linux is running on such system it
185automatically enables IOMMU if not enabled by the user already. These
186systems can be identified by reading ``1`` from
187``/sys/bus/thunderbolt/devices/domainX/iommu_dma_protection`` attribute.
188
189The driver does not do anything special in this case but because DMA
190protection is handled by the IOMMU, security levels (if set) are
191redundant. For this reason some systems ship with security level set to
192``none``. Other systems have security level set to ``user`` in order to
193support downgrade to older OS, so users who want to automatically
194authorize devices when IOMMU DMA protection is enabled can use the
195following ``udev`` rule::
196
197  ACTION=="add", SUBSYSTEM=="thunderbolt", ATTRS{iommu_dma_protection}=="1", ATTR{authorized}=="0", ATTR{authorized}="1"
198
199Upgrading NVM on Thunderbolt device, host or retimer
200----------------------------------------------------
201Since most of the functionality is handled in firmware running on a
202host controller or a device, it is important that the firmware can be
203upgraded to the latest where possible bugs in it have been fixed.
204Typically OEMs provide this firmware from their support site.
205
206There is also a central site which has links where to download firmware
207for some machines:
208
209  `Thunderbolt Updates <https://thunderbolttechnology.net/updates>`_
210
211Before you upgrade firmware on a device, host or retimer, please make
212sure it is a suitable upgrade. Failing to do that may render the device
213in a state where it cannot be used properly anymore without special
214tools!
215
216Host NVM upgrade on Apple Macs is not supported.
217
218Once the NVM image has been downloaded, you need to plug in a
219Thunderbolt device so that the host controller appears. It does not
220matter which device is connected (unless you are upgrading NVM on a
221device - then you need to connect that particular device).
222
223Note an OEM-specific method to power the controller up ("force power") may
224be available for your system in which case there is no need to plug in a
225Thunderbolt device.
226
227After that we can write the firmware to the non-active parts of the NVM
228of the host or device. As an example here is how Intel NUC6i7KYK (Skull
229Canyon) Thunderbolt controller NVM is upgraded::
230
231  # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem
232
233Once the operation completes we can trigger NVM authentication and
234upgrade process as follows::
235
236  # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate
237
238If no errors are returned, the host controller shortly disappears. Once
239it comes back the driver notices it and initiates a full power cycle.
240After a while the host controller appears again and this time it should
241be fully functional.
242
243We can verify that the new NVM firmware is active by running the following
244commands::
245
246  # cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate
247  0x0
248  # cat /sys/bus/thunderbolt/devices/0-0/nvm_version
249  18.0
250
251If ``nvm_authenticate`` contains anything other than 0x0 it is the error
252code from the last authentication cycle, which means the authentication
253of the NVM image failed.
254
255Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN``
256depend on the order they are registered in the NVMem subsystem. N in
257the name is the identifier added by the NVMem subsystem.
258
259Upgrading NVM when host controller is in safe mode
260--------------------------------------------------
261If the existing NVM is not properly authenticated (or is missing) the
262host controller goes into safe mode which means that the only available
263functionality is flashing a new NVM image. When in this mode, reading
264``nvm_version`` fails with ``ENODATA`` and the device identification
265information is missing.
266
267To recover from this mode, one needs to flash a valid NVM image to the
268host controller in the same way it is done in the previous chapter.
269
270Networking over Thunderbolt cable
271---------------------------------
272Thunderbolt technology allows software communication between two hosts
273connected by a Thunderbolt cable.
274
275It is possible to tunnel any kind of traffic over a Thunderbolt link but
276currently we only support Apple ThunderboltIP protocol.
277
278If the other host is running Windows or macOS, the only thing you need to
279do is to connect a Thunderbolt cable between the two hosts; the
280``thunderbolt-net`` driver is loaded automatically. If the other host is
281also Linux you should load ``thunderbolt-net`` manually on one host (it
282does not matter which one)::
283
284  # modprobe thunderbolt-net
285
286This triggers module load on the other host automatically. If the driver
287is built-in to the kernel image, there is no need to do anything.
288
289The driver will create one virtual ethernet interface per Thunderbolt
290port which are named like ``thunderbolt0`` and so on. From this point
291you can either use standard userspace tools like ``ifconfig`` to
292configure the interface or let your GUI handle it automatically.
293
294Forcing power
295-------------
296Many OEMs include a method that can be used to force the power of a
297Thunderbolt controller to an "On" state even if nothing is connected.
298If supported by your machine this will be exposed by the WMI bus with
299a sysfs attribute called "force_power".
300
301For example the intel-wmi-thunderbolt driver exposes this attribute in:
302  /sys/bus/wmi/devices/86CCFD48-205E-4A77-9C48-2021CBEDE341/force_power
303
304  To force the power to on, write 1 to this attribute file.
305  To disable force power, write 0 to this attribute file.
306
307Note: it's currently not possible to query the force power state of a platform.
308