1.. _cpu-topology-s390x:
2
3CPU topology on s390x
4=====================
5
6Since QEMU 8.2, CPU topology on s390x provides up to 3 levels of
7topology containers: drawers, books and sockets. They define a
8tree-shaped hierarchy.
9
10The socket container has one or more CPU entries.
11Each of these CPU entries consists of a bitmap and three CPU attributes:
12
13- CPU type
14- entitlement
15- dedication
16
17Each bit set in the bitmap correspond to a core-id of a vCPU with matching
18attributes.
19
20This documentation provides general information on S390 CPU topology,
21how to enable it and explains the new CPU attributes.
22For information on how to modify the S390 CPU topology and how to
23monitor polarization changes, see ``docs/devel/s390-cpu-topology.rst``.
24
25Prerequisites
26-------------
27
28To use the CPU topology, you need to run with KVM on a s390x host that
29uses the Linux kernel v6.0 or newer (which provide the so-called
30``KVM_CAP_S390_CPU_TOPOLOGY`` capability that allows QEMU to signal the
31CPU topology facility via the so-called STFLE bit 11 to the VM).
32
33Enabling CPU topology
34---------------------
35
36Currently, CPU topology is only enabled in the host model by default.
37
38Enabling CPU topology in a CPU model is done by setting the CPU flag
39``ctop`` to ``on`` as in:
40
41.. code-block:: bash
42
43   -cpu gen16b,ctop=on
44
45Having the topology disabled by default allows migration between
46old and new QEMU without adding new flags.
47
48Default topology usage
49----------------------
50
51The CPU topology can be specified on the QEMU command line
52with the ``-smp`` or the ``-device`` QEMU command arguments.
53
54Note also that since 7.2 threads are no longer supported in the topology
55and the ``-smp`` command line argument accepts only ``threads=1``.
56
57If none of the containers attributes (drawers, books, sockets) are
58specified for the ``-smp`` flag, the number of these containers
59is 1.
60
61Thus the following two options will result in the same topology:
62
63.. code-block:: bash
64
65    -smp cpus=5,drawer=1,books=1,sockets=8,cores=4,maxcpus=32
66
67and
68
69.. code-block:: bash
70
71    -smp cpus=5,sockets=8,cores=4,maxcpus=32
72
73When a CPU is defined by the ``-smp`` command argument, its position
74inside the topology is calculated by adding the CPUs to the topology
75based on the core-id starting with core-0 at position 0 of socket-0,
76book-0, drawer-0 and filling all CPUs of socket-0 before filling socket-1
77of book-0 and so on up to the last socket of the last book of the last
78drawer.
79
80When a CPU is defined by the ``-device`` command argument, the
81tree topology attributes must all be defined or all not defined.
82
83.. code-block:: bash
84
85    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1
86
87or
88
89.. code-block:: bash
90
91    -device gen16b-s390x-cpu,core-id=1,dedicated=true
92
93If none of the tree attributes (drawer, book, sockets), are specified
94for the ``-device`` argument, like for all CPUs defined with the ``-smp``
95command argument the topology tree attributes will be set by simply
96adding the CPUs to the topology based on the core-id.
97
98QEMU will not try to resolve collisions and will report an error if the
99CPU topology defined explicitly or implicitly on a ``-device``
100argument collides with the definition of a CPU implicitly defined
101on the ``-smp`` argument.
102
103When the topology modifier attributes are not defined for the
104``-device`` command argument they takes following default values:
105
106- dedicated: ``false``
107- entitlement: ``medium``
108
109
110Hot plug
111++++++++
112
113New CPUs can be plugged using the device_add hmp command as in:
114
115.. code-block:: bash
116
117  (qemu) device_add gen16b-s390x-cpu,core-id=9
118
119The placement of the CPU is derived from the core-id as described above.
120
121The topology can of course also be fully defined:
122
123.. code-block:: bash
124
125    (qemu) device_add gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1
126
127
128Examples
129++++++++
130
131In the following machine we define 8 sockets with 4 cores each.
132
133.. code-block:: bash
134
135  $ qemu-system-s390x -m 2G \
136    -cpu gen16b,ctop=on \
137    -smp cpus=5,sockets=8,cores=4,maxcpus=32 \
138    -device host-s390x-cpu,core-id=14 \
139
140A new CPUs can be plugged using the device_add hmp command as before:
141
142.. code-block:: bash
143
144  (qemu) device_add gen16b-s390x-cpu,core-id=9
145
146The core-id defines the placement of the core in the topology by
147starting with core 0 in socket 0 up to maxcpus.
148
149In the example above:
150
151* There are 5 CPUs provided to the guest with the ``-smp`` command line
152  They will take the core-ids 0,1,2,3,4
153  As we have 4 cores in a socket, we have 4 CPUs provided
154  to the guest in socket 0, with core-ids 0,1,2,3.
155  The last CPU, with core-id 4, will be on socket 1.
156
157* the core with ID 14 provided by the ``-device`` command line will
158  be placed in socket 3, with core-id 14
159
160* the core with ID 9 provided by the ``device_add`` qmp command will
161  be placed in socket 2, with core-id 9
162
163
164Polarization, entitlement and dedication
165----------------------------------------
166
167Polarization
168++++++++++++
169
170The polarization affects how the CPUs of a shared host are utilized/distributed
171among guests.
172The guest determines the polarization by using the PTF instruction.
173
174Polarization defines two models of CPU provisioning: horizontal
175and vertical.
176
177The horizontal polarization is the default model on boot and after
178subsystem reset. When horizontal polarization is in effect all vCPUs should
179have about equal resource provisioning.
180
181In the vertical polarization model vCPUs are unequal, but overall more resources
182might be available.
183The guest can make use of the vCPU entitlement information provided by the host
184to optimize kernel thread scheduling.
185
186A subsystem reset puts all vCPU of the configuration into the
187horizontal polarization.
188
189Entitlement
190+++++++++++
191
192The vertical polarization specifies that the guest's vCPU can get
193different real CPU provisioning:
194
195- a vCPU with vertical high entitlement specifies that this
196  vCPU gets 100% of the real CPU provisioning.
197
198- a vCPU with vertical medium entitlement specifies that this
199  vCPU shares the real CPU with other vCPUs.
200
201- a vCPU with vertical low entitlement specifies that this
202  vCPU only gets real CPU provisioning when no other vCPUs needs it.
203
204In the case a vCPU with vertical high entitlement does not use
205the real CPU, the unused "slack" can be dispatched to other vCPU
206with medium or low entitlement.
207
208A vCPU can be "dedicated" in which case the vCPU is fully dedicated to a single
209real CPU.
210
211The dedicated bit is an indication of affinity of a vCPU for a real CPU
212while the entitlement indicates the sharing or exclusivity of use.
213
214Defining the topology on the command line
215-----------------------------------------
216
217The topology can entirely be defined using -device cpu statements,
218with the exception of CPU 0 which must be defined with the -smp
219argument.
220
221For example, here we set the position of the cores 1,2,3 to
222drawer 1, book 1, socket 2 and cores 0,9 and 14 to drawer 0,
223book 0, socket 0 without defining entitlement or dedication.
224Core 4 will be set on its default position on socket 1
225(since we have 4 core per socket) and we define it as dedicated and
226with vertical high entitlement.
227
228.. code-block:: bash
229
230  $ qemu-system-s390x -m 2G \
231    -cpu gen16b,ctop=on \
232    -smp cpus=1,sockets=8,cores=4,maxcpus=32 \
233    \
234    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \
235    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \
236    -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \
237    \
238    -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \
239    -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \
240    \
241    -device gen16b-s390x-cpu,core-id=4,dedicated=on,entitlement=high
242
243The entitlement defined for the CPU 4 will only be used after the guest
244successfully enables vertical polarization by using the PTF instruction.
245