1.. SPDX-License-Identifier: GPL-2.0
2
3============
4x86 Topology
5============
6
7This documents and clarifies the main aspects of x86 topology modelling and
8representation in the kernel. Update/change when doing changes to the
9respective code.
10
11The architecture-agnostic topology definitions are in
12Documentation/admin-guide/cputopology.rst. This file holds x86-specific
13differences/specialities which must not necessarily apply to the generic
14definitions. Thus, the way to read up on Linux topology on x86 is to start
15with the generic one and look at this one in parallel for the x86 specifics.
16
17Needless to say, code should use the generic functions - this file is *only*
18here to *document* the inner workings of x86 topology.
19
20Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>.
21
22The main aim of the topology facilities is to present adequate interfaces to
23code which needs to know/query/use the structure of the running system wrt
24threads, cores, packages, etc.
25
26The kernel does not care about the concept of physical sockets because a
27socket has no relevance to software. It's an electromechanical component. In
28the past a socket always contained a single package (see below), but with the
29advent of Multi Chip Modules (MCM) a socket can hold more than one package. So
30there might be still references to sockets in the code, but they are of
31historical nature and should be cleaned up.
32
33The topology of a system is described in the units of:
34
35    - packages
36    - cores
37    - threads
38
39Package
40=======
41Packages contain a number of cores plus shared resources, e.g. DRAM
42controller, shared caches etc.
43
44Modern systems may also use the term 'Die' for package.
45
46AMD nomenclature for package is 'Node'.
47
48Package-related topology information in the kernel:
49
50  - cpuinfo_x86.x86_max_cores:
51
52    The number of cores in a package. This information is retrieved via CPUID.
53
54  - cpuinfo_x86.x86_max_dies:
55
56    The number of dies in a package. This information is retrieved via CPUID.
57
58  - cpuinfo_x86.cpu_die_id:
59
60    The physical ID of the die. This information is retrieved via CPUID.
61
62  - cpuinfo_x86.phys_proc_id:
63
64    The physical ID of the package. This information is retrieved via CPUID
65    and deduced from the APIC IDs of the cores in the package.
66
67    Modern systems use this value for the socket. There may be multiple
68    packages within a socket. This value may differ from cpu_die_id.
69
70  - cpuinfo_x86.logical_proc_id:
71
72    The logical ID of the package. As we do not trust BIOSes to enumerate the
73    packages in a consistent way, we introduced the concept of logical package
74    ID so we can sanely calculate the number of maximum possible packages in
75    the system and have the packages enumerated linearly.
76
77  - topology_max_packages():
78
79    The maximum possible number of packages in the system. Helpful for per
80    package facilities to preallocate per package information.
81
82  - cpu_llc_id:
83
84    A per-CPU variable containing:
85
86      - On Intel, the first APIC ID of the list of CPUs sharing the Last Level
87        Cache
88
89      - On AMD, the Node ID or Core Complex ID containing the Last Level
90        Cache. In general, it is a number identifying an LLC uniquely on the
91        system.
92
93Cores
94=====
95A core consists of 1 or more threads. It does not matter whether the threads
96are SMT- or CMT-type threads.
97
98AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
99"core".
100
101Core-related topology information in the kernel:
102
103  - smp_num_siblings:
104
105    The number of threads in a core. The number of threads in a package can be
106    calculated by::
107
108	threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings
109
110
111Threads
112=======
113A thread is a single scheduling unit. It's the equivalent to a logical Linux
114CPU.
115
116AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always
117uses "thread".
118
119Thread-related topology information in the kernel:
120
121  - topology_core_cpumask():
122
123    The cpumask contains all online threads in the package to which a thread
124    belongs.
125
126    The number of online threads is also printed in /proc/cpuinfo "siblings."
127
128  - topology_sibling_cpumask():
129
130    The cpumask contains all online threads in the core to which a thread
131    belongs.
132
133  - topology_logical_package_id():
134
135    The logical package ID to which a thread belongs.
136
137  - topology_physical_package_id():
138
139    The physical package ID to which a thread belongs.
140
141  - topology_core_id();
142
143    The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
144    "core_id."
145
146
147
148System topology examples
149========================
150
151.. note::
152  The alternative Linux CPU enumeration depends on how the BIOS enumerates the
153  threads. Many BIOSes enumerate all threads 0 first and then all threads 1.
154  That has the "advantage" that the logical Linux CPU numbers of threads 0 stay
155  the same whether threads are enabled or not. That's merely an implementation
156  detail and has no practical impact.
157
1581) Single Package, Single Core::
159
160   [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
161
1622) Single Package, Dual Core
163
164   a) One thread per core::
165
166	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
167		    -> [core 1] -> [thread 0] -> Linux CPU 1
168
169   b) Two threads per core::
170
171	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
172				-> [thread 1] -> Linux CPU 1
173		    -> [core 1] -> [thread 0] -> Linux CPU 2
174				-> [thread 1] -> Linux CPU 3
175
176      Alternative enumeration::
177
178	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
179				-> [thread 1] -> Linux CPU 2
180		    -> [core 1] -> [thread 0] -> Linux CPU 1
181				-> [thread 1] -> Linux CPU 3
182
183      AMD nomenclature for CMT systems::
184
185	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
186				     -> [Compute Unit Core 1] -> Linux CPU 1
187		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
188				     -> [Compute Unit Core 1] -> Linux CPU 3
189
1904) Dual Package, Dual Core
191
192   a) One thread per core::
193
194	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
195		    -> [core 1] -> [thread 0] -> Linux CPU 1
196
197	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
198		    -> [core 1] -> [thread 0] -> Linux CPU 3
199
200   b) Two threads per core::
201
202	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
203				-> [thread 1] -> Linux CPU 1
204		    -> [core 1] -> [thread 0] -> Linux CPU 2
205				-> [thread 1] -> Linux CPU 3
206
207	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 4
208				-> [thread 1] -> Linux CPU 5
209		    -> [core 1] -> [thread 0] -> Linux CPU 6
210				-> [thread 1] -> Linux CPU 7
211
212      Alternative enumeration::
213
214	[package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
215				-> [thread 1] -> Linux CPU 4
216		    -> [core 1] -> [thread 0] -> Linux CPU 1
217				-> [thread 1] -> Linux CPU 5
218
219	[package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
220				-> [thread 1] -> Linux CPU 6
221		    -> [core 1] -> [thread 0] -> Linux CPU 3
222				-> [thread 1] -> Linux CPU 7
223
224      AMD nomenclature for CMT systems::
225
226	[node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
227				     -> [Compute Unit Core 1] -> Linux CPU 1
228		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
229				     -> [Compute Unit Core 1] -> Linux CPU 3
230
231	[node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4
232				     -> [Compute Unit Core 1] -> Linux CPU 5
233		 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6
234				     -> [Compute Unit Core 1] -> Linux CPU 7
235