admin-guide/perf/nvidia-pmu.rst

2 NVIDIA Tegra SoC Uncore Performance Monitoring Unit (PMU)
5 The NVIDIA Tegra SoC includes various system PMUs to measure key performance
9 * NVLink-C2C0
10 * NVLink-C2C1
15 ----------
19 PMUs are managed by a common driver "arm-cs-arch-pmu". This driver describes
29 -------
32 strongly-ordered (SO) PCIE write traffic to local/remote memory. Please see
37 see /sys/bus/event_sources/devices/nvidia_scf_pmu_<socket-id>.
43    perf stat -a -e nvidia_scf_pmu_0/event=0x0/
47    perf stat -a -e nvidia_scf_pmu_1/event=0x0/
49 NVLink-C2C0 PMU
50 --------------------
52 The NVLink-C2C0 PMU monitors incoming traffic from a GPU/CPU connected with
53 NVLink-C2C (Chip-2-Chip) interconnect. The type of traffic captured by this PMU
54 varies dependent on the chip configuration:
56 * NVIDIA Grace Hopper Superchip: Hopper GPU is connected with Grace SoC.
63   PCIE device of the remote SoC.
69 see /sys/bus/event_sources/devices/nvidia_nvlink_c2c0_pmu_<socket-id>.
75    perf stat -a -e nvidia_nvlink_c2c0_pmu_0/event=0x0/
79    perf stat -a -e nvidia_nvlink_c2c0_pmu_1/event=0x0/
83    perf stat -a -e nvidia_nvlink_c2c0_pmu_2/event=0x0/
87    perf stat -a -e nvidia_nvlink_c2c0_pmu_3/event=0x0/
89 NVLink-C2C1 PMU
90 -------------------
92 The NVLink-C2C1 PMU monitors incoming traffic from a GPU connected with
93 NVLink-C2C (Chip-2-Chip) interconnect. This PMU captures untranslated GPU
94 traffic, in contrast with NvLink-C2C0 PMU that captures ATS translated traffic.
99 see /sys/bus/event_sources/devices/nvidia_nvlink_c2c1_pmu_<socket-id>.
105    perf stat -a -e nvidia_nvlink_c2c1_pmu_0/event=0x0/
109    perf stat -a -e nvidia_nvlink_c2c1_pmu_1/event=0x0/
113    perf stat -a -e nvidia_nvlink_c2c1_pmu_2/event=0x0/
117    perf stat -a -e nvidia_nvlink_c2c1_pmu_3/event=0x0/
120 ---------------
128 see /sys/bus/event_sources/devices/nvidia_cnvlink_pmu_<socket-id>.
130 Each SoC socket can be connected to one or more sockets via CNVLink. The user can
134 /sys/bus/event_sources/devices/nvidia_cnvlink_pmu_<socket-id>/format/rem_socket
145    perf stat -a -e nvidia_cnvlink_pmu_0/event=0x0,rem_socket=0xE/
149    perf stat -a -e nvidia_cnvlink_pmu_1/event=0x0,rem_socket=0xD/
153    perf stat -a -e nvidia_cnvlink_pmu_2/event=0x0,rem_socket=0xB/
157    perf stat -a -e nvidia_cnvlink_pmu_3/event=0x0,rem_socket=0x7/
161 ------------
168 see /sys/bus/event_sources/devices/nvidia_pcie_pmu_<socket-id>.
170 Each SoC socket can support multiple root ports. The user can use
173 /sys/bus/event_sources/devices/nvidia_pcie_pmu_<socket-id>/format/root_port
180    perf stat -a -e nvidia_pcie_pmu_0/event=0x0,root_port=0x3/
184    perf stat -a -e nvidia_pcie_pmu_1/event=0x0,root_port=0x3/
189 ----------------
191 The PMU traffic coverage may vary dependent on the chip configuration:
193 * **NVIDIA Grace Hopper Superchip**: Hopper GPU is connected with Grace SoC.
198    * SOCKET-A                      *          * SOCKET-B                      *
207    *  : GPU :<--NVLink-->: Grace :<---CNVLink--->: Grace :<--NVLink-->: GPU : *
208    *  :     :    C2C     :  SoC  : *          *  :  SoC  :    C2C     :     : *
222   | Following table contains traffic coverage of Grace SoC PMU in socket-A:
226    +--------------+-------+-----------+-----------+-----+----------+----------+
228    +              +-------+-----------+-----------+-----+----------+----------+
229    | Destination  |       |GPU ATS    |GPU Not-ATS|     | Socket-B | Socket-B |
233    | Local        | PCIE  |NVLink-C2C0|NVLink-C2C1| SCF | SCF PMU  | CNVLink  |
235    +--------------+-------+-----------+-----------+-----+----------+----------+
236    | Local GMEM   | PCIE  |    N/A    |NVLink-C2C1| SCF | SCF PMU  | CNVLink  |
238    +--------------+-------+-----------+-----------+-----+----------+----------+
239    | Remote       | PCIE  |NVLink-C2C0|NVLink-C2C1| SCF |          |          |
242    +--------------+-------+-----------+-----------+-----+----------+----------+
243    | Remote GMEM  | PCIE  |NVLink-C2C0|NVLink-C2C1| SCF |          |          |
245    +--------------+-------+-----------+-----------+-----+----------+----------+
255    * SOCKET-A        *             * SOCKET-B        *
264    *    : Grace :<--------NVLink------->: Grace :    *
265    *    :  SoC  :    *     C2C     *    :  SoC  :    *
279   | Following table contains traffic coverage of Grace SoC PMU in socket-A:
283    +-----------------+-----------+---------+----------+-------------+
285    +                 +-----------+---------+----------+-------------+
286    | Destination     |           |         | Socket-B | Socket-B    |
290    | Local           |  PCIE PMU | SCF PMU | SCF PMU  | NVLink-C2C0 |
292    +-----------------+-----------+---------+----------+-------------+
295    | over NVLink-C2C |           |         |          |             |
296    +-----------------+-----------+---------+----------+-------------+