1=========================================================
2NVIDIA Tegra SoC Uncore Performance Monitoring Unit (PMU)
3=========================================================
4
5The NVIDIA Tegra SoC includes various system PMUs to measure key performance
6metrics like memory bandwidth, latency, and utilization:
7
8* Scalable Coherency Fabric (SCF)
9* NVLink-C2C0
10* NVLink-C2C1
11* CNVLink
12* PCIE
13
14PMU Driver
15----------
16
17The PMUs in this document are based on ARM CoreSight PMU Architecture as
18described in document: ARM IHI 0091. Since this is a standard architecture, the
19PMUs are managed by a common driver "arm-cs-arch-pmu". This driver describes
20the available events and configuration of each PMU in sysfs. Please see the
21sections below to get the sysfs path of each PMU. Like other uncore PMU drivers,
22the driver provides "cpumask" sysfs attribute to show the CPU id used to handle
23the PMU event. There is also "associated_cpus" sysfs attribute, which contains a
24list of CPUs associated with the PMU instance.
25
26.. _SCF_PMU_Section:
27
28SCF PMU
29-------
30
31The SCF PMU monitors system level cache events, CPU traffic, and
32strongly-ordered (SO) PCIE write traffic to local/remote memory. Please see
33:ref:`NVIDIA_Uncore_PMU_Traffic_Coverage_Section` for more info about the PMU
34traffic coverage.
35
36The events and configuration options of this PMU device are described in sysfs,
37see /sys/bus/event_sources/devices/nvidia_scf_pmu_<socket-id>.
38
39Example usage:
40
41* Count event id 0x0 in socket 0::
42
43   perf stat -a -e nvidia_scf_pmu_0/event=0x0/
44
45* Count event id 0x0 in socket 1::
46
47   perf stat -a -e nvidia_scf_pmu_1/event=0x0/
48
49NVLink-C2C0 PMU
50--------------------
51
52The NVLink-C2C0 PMU monitors incoming traffic from a GPU/CPU connected with
53NVLink-C2C (Chip-2-Chip) interconnect. The type of traffic captured by this PMU
54varies dependent on the chip configuration:
55
56* NVIDIA Grace Hopper Superchip: Hopper GPU is connected with Grace SoC.
57
58  In this config, the PMU captures GPU ATS translated or EGM traffic from the GPU.
59
60* NVIDIA Grace CPU Superchip: two Grace CPU SoCs are connected.
61
62  In this config, the PMU captures read and relaxed ordered (RO) writes from
63  PCIE device of the remote SoC.
64
65Please see :ref:`NVIDIA_Uncore_PMU_Traffic_Coverage_Section` for more info about
66the PMU traffic coverage.
67
68The events and configuration options of this PMU device are described in sysfs,
69see /sys/bus/event_sources/devices/nvidia_nvlink_c2c0_pmu_<socket-id>.
70
71Example usage:
72
73* Count event id 0x0 from the GPU/CPU connected with socket 0::
74
75   perf stat -a -e nvidia_nvlink_c2c0_pmu_0/event=0x0/
76
77* Count event id 0x0 from the GPU/CPU connected with socket 1::
78
79   perf stat -a -e nvidia_nvlink_c2c0_pmu_1/event=0x0/
80
81* Count event id 0x0 from the GPU/CPU connected with socket 2::
82
83   perf stat -a -e nvidia_nvlink_c2c0_pmu_2/event=0x0/
84
85* Count event id 0x0 from the GPU/CPU connected with socket 3::
86
87   perf stat -a -e nvidia_nvlink_c2c0_pmu_3/event=0x0/
88
89NVLink-C2C1 PMU
90-------------------
91
92The NVLink-C2C1 PMU monitors incoming traffic from a GPU connected with
93NVLink-C2C (Chip-2-Chip) interconnect. This PMU captures untranslated GPU
94traffic, in contrast with NvLink-C2C0 PMU that captures ATS translated traffic.
95Please see :ref:`NVIDIA_Uncore_PMU_Traffic_Coverage_Section` for more info about
96the PMU traffic coverage.
97
98The events and configuration options of this PMU device are described in sysfs,
99see /sys/bus/event_sources/devices/nvidia_nvlink_c2c1_pmu_<socket-id>.
100
101Example usage:
102
103* Count event id 0x0 from the GPU connected with socket 0::
104
105   perf stat -a -e nvidia_nvlink_c2c1_pmu_0/event=0x0/
106
107* Count event id 0x0 from the GPU connected with socket 1::
108
109   perf stat -a -e nvidia_nvlink_c2c1_pmu_1/event=0x0/
110
111* Count event id 0x0 from the GPU connected with socket 2::
112
113   perf stat -a -e nvidia_nvlink_c2c1_pmu_2/event=0x0/
114
115* Count event id 0x0 from the GPU connected with socket 3::
116
117   perf stat -a -e nvidia_nvlink_c2c1_pmu_3/event=0x0/
118
119CNVLink PMU
120---------------
121
122The CNVLink PMU monitors traffic from GPU and PCIE device on remote sockets
123to local memory. For PCIE traffic, this PMU captures read and relaxed ordered
124(RO) write traffic. Please see :ref:`NVIDIA_Uncore_PMU_Traffic_Coverage_Section`
125for more info about the PMU traffic coverage.
126
127The events and configuration options of this PMU device are described in sysfs,
128see /sys/bus/event_sources/devices/nvidia_cnvlink_pmu_<socket-id>.
129
130Each SoC socket can be connected to one or more sockets via CNVLink. The user can
131use "rem_socket" bitmap parameter to select the remote socket(s) to monitor.
132Each bit represents the socket number, e.g. "rem_socket=0xE" corresponds to
133socket 1 to 3.
134/sys/bus/event_sources/devices/nvidia_cnvlink_pmu_<socket-id>/format/rem_socket
135shows the valid bits that can be set in the "rem_socket" parameter.
136
137The PMU can not distinguish the remote traffic initiator, therefore it does not
138provide filter to select the traffic source to monitor. It reports combined
139traffic from remote GPU and PCIE devices.
140
141Example usage:
142
143* Count event id 0x0 for the traffic from remote socket 1, 2, and 3 to socket 0::
144
145   perf stat -a -e nvidia_cnvlink_pmu_0/event=0x0,rem_socket=0xE/
146
147* Count event id 0x0 for the traffic from remote socket 0, 2, and 3 to socket 1::
148
149   perf stat -a -e nvidia_cnvlink_pmu_1/event=0x0,rem_socket=0xD/
150
151* Count event id 0x0 for the traffic from remote socket 0, 1, and 3 to socket 2::
152
153   perf stat -a -e nvidia_cnvlink_pmu_2/event=0x0,rem_socket=0xB/
154
155* Count event id 0x0 for the traffic from remote socket 0, 1, and 2 to socket 3::
156
157   perf stat -a -e nvidia_cnvlink_pmu_3/event=0x0,rem_socket=0x7/
158
159
160PCIE PMU
161------------
162
163The PCIE PMU monitors all read/write traffic from PCIE root ports to
164local/remote memory. Please see :ref:`NVIDIA_Uncore_PMU_Traffic_Coverage_Section`
165for more info about the PMU traffic coverage.
166
167The events and configuration options of this PMU device are described in sysfs,
168see /sys/bus/event_sources/devices/nvidia_pcie_pmu_<socket-id>.
169
170Each SoC socket can support multiple root ports. The user can use
171"root_port" bitmap parameter to select the port(s) to monitor, i.e.
172"root_port=0xF" corresponds to root port 0 to 3.
173/sys/bus/event_sources/devices/nvidia_pcie_pmu_<socket-id>/format/root_port
174shows the valid bits that can be set in the "root_port" parameter.
175
176Example usage:
177
178* Count event id 0x0 from root port 0 and 1 of socket 0::
179
180   perf stat -a -e nvidia_pcie_pmu_0/event=0x0,root_port=0x3/
181
182* Count event id 0x0 from root port 0 and 1 of socket 1::
183
184   perf stat -a -e nvidia_pcie_pmu_1/event=0x0,root_port=0x3/
185
186.. _NVIDIA_Uncore_PMU_Traffic_Coverage_Section:
187
188Traffic Coverage
189----------------
190
191The PMU traffic coverage may vary dependent on the chip configuration:
192
193* **NVIDIA Grace Hopper Superchip**: Hopper GPU is connected with Grace SoC.
194
195  Example configuration with two Grace SoCs::
196
197   *********************************          *********************************
198   * SOCKET-A                      *          * SOCKET-B                      *
199   *                               *          *                               *
200   *                     ::::::::  *          *  ::::::::                     *
201   *                     : PCIE :  *          *  : PCIE :                     *
202   *                     ::::::::  *          *  ::::::::                     *
203   *                         |     *          *      |                        *
204   *                         |     *          *      |                        *
205   *  :::::::            ::::::::: *          *  :::::::::            ::::::: *
206   *  :     :            :       : *          *  :       :            :     : *
207   *  : GPU :<--NVLink-->: Grace :<---CNVLink--->: Grace :<--NVLink-->: GPU : *
208   *  :     :    C2C     :  SoC  : *          *  :  SoC  :    C2C     :     : *
209   *  :::::::            ::::::::: *          *  :::::::::            ::::::: *
210   *     |                   |     *          *      |                   |    *
211   *     |                   |     *          *      |                   |    *
212   *  &&&&&&&&           &&&&&&&&  *          *   &&&&&&&&           &&&&&&&& *
213   *  & GMEM &           & CMEM &  *          *   & CMEM &           & GMEM & *
214   *  &&&&&&&&           &&&&&&&&  *          *   &&&&&&&&           &&&&&&&& *
215   *                               *          *                               *
216   *********************************          *********************************
217
218   GMEM = GPU Memory (e.g. HBM)
219   CMEM = CPU Memory (e.g. LPDDR5X)
220
221  |
222  | Following table contains traffic coverage of Grace SoC PMU in socket-A:
223
224  ::
225
226   +--------------+-------+-----------+-----------+-----+----------+----------+
227   |              |                        Source                             |
228   +              +-------+-----------+-----------+-----+----------+----------+
229   | Destination  |       |GPU ATS    |GPU Not-ATS|     | Socket-B | Socket-B |
230   |              |PCI R/W|Translated,|Translated | CPU | CPU/PCIE1| GPU/PCIE2|
231   |              |       |EGM        |           |     |          |          |
232   +==============+=======+===========+===========+=====+==========+==========+
233   | Local        | PCIE  |NVLink-C2C0|NVLink-C2C1| SCF | SCF PMU  | CNVLink  |
234   | SYSRAM/CMEM  | PMU   |PMU        |PMU        | PMU |          | PMU      |
235   +--------------+-------+-----------+-----------+-----+----------+----------+
236   | Local GMEM   | PCIE  |    N/A    |NVLink-C2C1| SCF | SCF PMU  | CNVLink  |
237   |              | PMU   |           |PMU        | PMU |          | PMU      |
238   +--------------+-------+-----------+-----------+-----+----------+----------+
239   | Remote       | PCIE  |NVLink-C2C0|NVLink-C2C1| SCF |          |          |
240   | SYSRAM/CMEM  | PMU   |PMU        |PMU        | PMU |   N/A    |   N/A    |
241   | over CNVLink |       |           |           |     |          |          |
242   +--------------+-------+-----------+-----------+-----+----------+----------+
243   | Remote GMEM  | PCIE  |NVLink-C2C0|NVLink-C2C1| SCF |          |          |
244   | over CNVLink | PMU   |PMU        |PMU        | PMU |   N/A    |   N/A    |
245   +--------------+-------+-----------+-----------+-----+----------+----------+
246
247   PCIE1 traffic represents strongly ordered (SO) writes.
248   PCIE2 traffic represents reads and relaxed ordered (RO) writes.
249
250* **NVIDIA Grace CPU Superchip**: two Grace CPU SoCs are connected.
251
252  Example configuration with two Grace SoCs::
253
254   *******************             *******************
255   * SOCKET-A        *             * SOCKET-B        *
256   *                 *             *                 *
257   *    ::::::::     *             *    ::::::::     *
258   *    : PCIE :     *             *    : PCIE :     *
259   *    ::::::::     *             *    ::::::::     *
260   *        |        *             *        |        *
261   *        |        *             *        |        *
262   *    :::::::::    *             *    :::::::::    *
263   *    :       :    *             *    :       :    *
264   *    : Grace :<--------NVLink------->: Grace :    *
265   *    :  SoC  :    *     C2C     *    :  SoC  :    *
266   *    :::::::::    *             *    :::::::::    *
267   *        |        *             *        |        *
268   *        |        *             *        |        *
269   *     &&&&&&&&    *             *     &&&&&&&&    *
270   *     & CMEM &    *             *     & CMEM &    *
271   *     &&&&&&&&    *             *     &&&&&&&&    *
272   *                 *             *                 *
273   *******************             *******************
274
275   GMEM = GPU Memory (e.g. HBM)
276   CMEM = CPU Memory (e.g. LPDDR5X)
277
278  |
279  | Following table contains traffic coverage of Grace SoC PMU in socket-A:
280
281  ::
282
283   +-----------------+-----------+---------+----------+-------------+
284   |                 |                      Source                  |
285   +                 +-----------+---------+----------+-------------+
286   | Destination     |           |         | Socket-B | Socket-B    |
287   |                 |  PCI R/W  |   CPU   | CPU/PCIE1| PCIE2       |
288   |                 |           |         |          |             |
289   +=================+===========+=========+==========+=============+
290   | Local           |  PCIE PMU | SCF PMU | SCF PMU  | NVLink-C2C0 |
291   | SYSRAM/CMEM     |           |         |          | PMU         |
292   +-----------------+-----------+---------+----------+-------------+
293   | Remote          |           |         |          |             |
294   | SYSRAM/CMEM     |  PCIE PMU | SCF PMU |   N/A    |     N/A     |
295   | over NVLink-C2C |           |         |          |             |
296   +-----------------+-----------+---------+----------+-------------+
297
298   PCIE1 traffic represents strongly ordered (SO) writes.
299   PCIE2 traffic represents reads and relaxed ordered (RO) writes.
300