xref: /openbmc/linux/Documentation/fpga/dfl.rst (revision c8ed9fc9) 
1=================================================
2FPGA Device Feature List (DFL) Framework Overview
3=================================================
4
5Authors:
6
7- Enno Luebbers <enno.luebbers@intel.com>
8- Xiao Guangrong <guangrong.xiao@linux.intel.com>
9- Wu Hao <hao.wu@intel.com>
10
11The Device Feature List (DFL) FPGA framework (and drivers according to this
12this framework) hides the very details of low layer hardwares and provides
13unified interfaces to userspace. Applications could use these interfaces to
14configure, enumerate, open and access FPGA accelerators on platforms which
15implement the DFL in the device memory. Besides this, the DFL framework
16enables system level management functions such as FPGA reconfiguration.
17
18
19Device Feature List (DFL) Overview
20==================================
21Device Feature List (DFL) defines a linked list of feature headers within the
22device MMIO space to provide an extensible way of adding features. Software can
23walk through these predefined data structures to enumerate FPGA features:
24FPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features,
25as illustrated below::
26
27    Header            Header            Header            Header
28 +----------+  +-->+----------+  +-->+----------+  +-->+----------+
29 |   Type   |  |   |  Type    |  |   |  Type    |  |   |  Type    |
30 |   FIU    |  |   | Private  |  |   | Private  |  |   | Private  |
31 +----------+  |   | Feature  |  |   | Feature  |  |   | Feature  |
32 | Next_DFH |--+   +----------+  |   +----------+  |   +----------+
33 +----------+      | Next_DFH |--+   | Next_DFH |--+   | Next_DFH |--> NULL
34 |    ID    |      +----------+      +----------+      +----------+
35 +----------+      |    ID    |      |    ID    |      |    ID    |
36 | Next_AFU |--+   +----------+      +----------+      +----------+
37 +----------+  |   | Feature  |      | Feature  |      | Feature  |
38 |  Header  |  |   | Register |      | Register |      | Register |
39 | Register |  |   |   Set    |      |   Set    |      |   Set    |
40 |   Set    |  |   +----------+      +----------+      +----------+
41 +----------+  |      Header
42               +-->+----------+
43                   |   Type   |
44                   |   AFU    |
45                   +----------+
46                   | Next_DFH |--> NULL
47                   +----------+
48                   |   GUID   |
49                   +----------+
50                   |  Header  |
51                   | Register |
52                   |   Set    |
53                   +----------+
54
55FPGA Interface Unit (FIU) represents a standalone functional unit for the
56interface to FPGA, e.g. the FPGA Management Engine (FME) and Port (more
57descriptions on FME and Port in later sections).
58
59Accelerated Function Unit (AFU) represents a FPGA programmable region and
60always connects to a FIU (e.g. a Port) as its child as illustrated above.
61
62Private Features represent sub features of the FIU and AFU. They could be
63various function blocks with different IDs, but all private features which
64belong to the same FIU or AFU, must be linked to one list via the Next Device
65Feature Header (Next_DFH) pointer.
66
67Each FIU, AFU and Private Feature could implement its own functional registers.
68The functional register set for FIU and AFU, is named as Header Register Set,
69e.g. FME Header Register Set, and the one for Private Feature, is named as
70Feature Register Set, e.g. FME Partial Reconfiguration Feature Register Set.
71
72This Device Feature List provides a way of linking features together, it's
73convenient for software to locate each feature by walking through this list,
74and can be implemented in register regions of any FPGA device.
75
76
77FIU - FME (FPGA Management Engine)
78==================================
79The FPGA Management Engine performs reconfiguration and other infrastructure
80functions. Each FPGA device only has one FME.
81
82User-space applications can acquire exclusive access to the FME using open(),
83and release it using close().
84
85The following functions are exposed through ioctls:
86
87- Get driver API version (DFL_FPGA_GET_API_VERSION)
88- Check for extensions (DFL_FPGA_CHECK_EXTENSION)
89- Program bitstream (DFL_FPGA_FME_PORT_PR)
90- Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN)
91- Release port from PF (DFL_FPGA_FME_PORT_RELEASE)
92
93More functions are exposed through sysfs
94(/sys/class/fpga_region/regionX/dfl-fme.n/):
95
96 Read bitstream ID (bitstream_id)
97     bitstream_id indicates version of the static FPGA region.
98
99 Read bitstream metadata (bitstream_metadata)
100     bitstream_metadata includes detailed information of static FPGA region,
101     e.g. synthesis date and seed.
102
103 Read number of ports (ports_num)
104     one FPGA device may have more than one port, this sysfs interface indicates
105     how many ports the FPGA device has.
106
107 Global error reporting management (errors/)
108     error reporting sysfs interfaces allow user to read errors detected by the
109     hardware, and clear the logged errors.
110
111 Power management (dfl_fme_power hwmon)
112     power management hwmon sysfs interfaces allow user to read power management
113     information (power consumption, thresholds, threshold status, limits, etc.)
114     and configure power thresholds for different throttling levels.
115
116 Thermal management (dfl_fme_thermal hwmon)
117     thermal management hwmon sysfs interfaces allow user to read thermal
118     management information (current temperature, thresholds, threshold status,
119     etc.).
120
121 Performance reporting
122     performance counters are exposed through perf PMU APIs. Standard perf tool
123     can be used to monitor all available perf events. Please see performance
124     counter section below for more detailed information.
125
126
127FIU - PORT
128==========
129A port represents the interface between the static FPGA fabric and a partially
130reconfigurable region containing an AFU. It controls the communication from SW
131to the accelerator and exposes features such as reset and debug. Each FPGA
132device may have more than one port, but always one AFU per port.
133
134
135AFU
136===
137An AFU is attached to a port FIU and exposes a fixed length MMIO region to be
138used for accelerator-specific control registers.
139
140User-space applications can acquire exclusive access to an AFU attached to a
141port by using open() on the port device node and release it using close().
142
143The following functions are exposed through ioctls:
144
145- Get driver API version (DFL_FPGA_GET_API_VERSION)
146- Check for extensions (DFL_FPGA_CHECK_EXTENSION)
147- Get port info (DFL_FPGA_PORT_GET_INFO)
148- Get MMIO region info (DFL_FPGA_PORT_GET_REGION_INFO)
149- Map DMA buffer (DFL_FPGA_PORT_DMA_MAP)
150- Unmap DMA buffer (DFL_FPGA_PORT_DMA_UNMAP)
151- Reset AFU (DFL_FPGA_PORT_RESET)
152
153DFL_FPGA_PORT_RESET:
154  reset the FPGA Port and its AFU. Userspace can do Port
155  reset at any time, e.g. during DMA or Partial Reconfiguration. But it should
156  never cause any system level issue, only functional failure (e.g. DMA or PR
157  operation failure) and be recoverable from the failure.
158
159User-space applications can also mmap() accelerator MMIO regions.
160
161More functions are exposed through sysfs:
162(/sys/class/fpga_region/<regionX>/<dfl-port.m>/):
163
164 Read Accelerator GUID (afu_id)
165     afu_id indicates which PR bitstream is programmed to this AFU.
166
167 Error reporting (errors/)
168     error reporting sysfs interfaces allow user to read port/afu errors
169     detected by the hardware, and clear the logged errors.
170
171
172DFL Framework Overview
173======================
174
175::
176
177         +----------+    +--------+ +--------+ +--------+
178         |   FME    |    |  AFU   | |  AFU   | |  AFU   |
179         |  Module  |    | Module | | Module | | Module |
180         +----------+    +--------+ +--------+ +--------+
181                 +-----------------------+
182                 | FPGA Container Device |    Device Feature List
183                 |  (FPGA Base Region)   |         Framework
184                 +-----------------------+
185  ------------------------------------------------------------------
186               +----------------------------+
187               |   FPGA DFL Device Module   |
188               | (e.g. PCIE/Platform Device)|
189               +----------------------------+
190                 +------------------------+
191                 |  FPGA Hardware Device  |
192                 +------------------------+
193
194DFL framework in kernel provides common interfaces to create container device
195(FPGA base region), discover feature devices and their private features from the
196given Device Feature Lists and create platform devices for feature devices
197(e.g. FME, Port and AFU) with related resources under the container device. It
198also abstracts operations for the private features and exposes common ops to
199feature device drivers.
200
201The FPGA DFL Device could be different hardwares, e.g. PCIe device, platform
202device and etc. Its driver module is always loaded first once the device is
203created by the system. This driver plays an infrastructural role in the
204driver architecture. It locates the DFLs in the device memory, handles them
205and related resources to common interfaces from DFL framework for enumeration.
206(Please refer to drivers/fpga/dfl.c for detailed enumeration APIs).
207
208The FPGA Management Engine (FME) driver is a platform driver which is loaded
209automatically after FME platform device creation from the DFL device module. It
210provides the key features for FPGA management, including:
211
212	a) Expose static FPGA region information, e.g. version and metadata.
213	   Users can read related information via sysfs interfaces exposed
214	   by FME driver.
215
216	b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
217	   bridges and FPGA regions during PR sub feature initialization. Once
218	   it receives a DFL_FPGA_FME_PORT_PR ioctl from user, it invokes the
219	   common interface function from FPGA Region to complete the partial
220	   reconfiguration of the PR bitstream to the given port.
221
222Similar to the FME driver, the FPGA Accelerated Function Unit (AFU) driver is
223probed once the AFU platform device is created. The main function of this module
224is to provide an interface for userspace applications to access the individual
225accelerators, including basic reset control on port, AFU MMIO region export, dma
226buffer mapping service functions.
227
228After feature platform devices creation, matched platform drivers will be loaded
229automatically to handle different functionalities. Please refer to next sections
230for detailed information on functional units which have been already implemented
231under this DFL framework.
232
233
234Partial Reconfiguration
235=======================
236As mentioned above, accelerators can be reconfigured through partial
237reconfiguration of a PR bitstream file. The PR bitstream file must have been
238generated for the exact static FPGA region and targeted reconfigurable region
239(port) of the FPGA, otherwise, the reconfiguration operation will fail and
240possibly cause system instability. This compatibility can be checked by
241comparing the compatibility ID noted in the header of PR bitstream file against
242the compat_id exposed by the target FPGA region. This check is usually done by
243userspace before calling the reconfiguration IOCTL.
244
245
246FPGA virtualization - PCIe SRIOV
247================================
248This section describes the virtualization support on DFL based FPGA device to
249enable accessing an accelerator from applications running in a virtual machine
250(VM). This section only describes the PCIe based FPGA device with SRIOV support.
251
252Features supported by the particular FPGA device are exposed through Device
253Feature Lists, as illustrated below:
254
255::
256
257    +-------------------------------+  +-------------+
258    |              PF               |  |     VF      |
259    +-------------------------------+  +-------------+
260        ^            ^         ^              ^
261        |            |         |              |
262  +-----|------------|---------|--------------|-------+
263  |     |            |         |              |       |
264  |  +-----+     +-------+ +-------+      +-------+   |
265  |  | FME |     | Port0 | | Port1 |      | Port2 |   |
266  |  +-----+     +-------+ +-------+      +-------+   |
267  |                  ^         ^              ^       |
268  |                  |         |              |       |
269  |              +-------+ +------+       +-------+   |
270  |              |  AFU  | |  AFU |       |  AFU  |   |
271  |              +-------+ +------+       +-------+   |
272  |                                                   |
273  |            DFL based FPGA PCIe Device             |
274  +---------------------------------------------------+
275
276FME is always accessed through the physical function (PF).
277
278Ports (and related AFUs) are accessed via PF by default, but could be exposed
279through virtual function (VF) devices via PCIe SRIOV. Each VF only contains
2801 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)
281created via PCIe SRIOV interface, to virtual machines.
282
283The driver organization in virtualization case is illustrated below:
284::
285
286    +-------++------++------+             |
287    | FME   || FME  || FME  |             |
288    | FPGA  || FPGA || FPGA |             |
289    |Manager||Bridge||Region|             |
290    +-------++------++------+             |
291    +-----------------------+  +--------+ |             +--------+
292    |          FME          |  |  AFU   | |             |  AFU   |
293    |         Module        |  | Module | |             | Module |
294    +-----------------------+  +--------+ |             +--------+
295          +-----------------------+       |       +-----------------------+
296          | FPGA Container Device |       |       | FPGA Container Device |
297          |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |
298          +-----------------------+       |       +-----------------------+
299            +------------------+          |         +------------------+
300            | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
301            +------------------+   Host   | Machine +------------------+
302   -------------------------------------- | ------------------------------
303             +---------------+            |          +---------------+
304             | PCI PF Device |            |          | PCI VF Device |
305             +---------------+            |          +---------------+
306
307FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device
308is detected. It:
309
310* Finishes enumeration on both FPGA PCIe PF and VF device using common
311  interfaces from DFL framework.
312* Supports SRIOV.
313
314The FME device driver plays a management role in this driver architecture, it
315provides ioctls to release Port from PF and assign Port to PF. After release
316a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV
317sysfs interface.
318
319To enable accessing an accelerator from applications running in a VM, the
320respective AFU's port needs to be assigned to a VF using the following steps:
321
322#. The PF owns all AFU ports by default. Any port that needs to be
323   reassigned to a VF must first be released through the
324   DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device.
325
326#. Once N ports are released from PF, then user can use command below
327   to enable SRIOV and VFs. Each VF owns only one Port with AFU.
328
329   ::
330
331      echo N > $PCI_DEVICE_PATH/sriov_numvfs
332
333#. Pass through the VFs to VMs
334
335#. The AFU under VF is accessible from applications in VM (using the
336   same driver inside the VF).
337
338Note that an FME can't be assigned to a VF, thus PR and other management
339functions are only available via the PF.
340
341Device enumeration
342==================
343This section introduces how applications enumerate the fpga device from
344the sysfs hierarchy under /sys/class/fpga_region.
345
346In the example below, two DFL based FPGA devices are installed in the host. Each
347fpga device has one FME and two ports (AFUs).
348
349FPGA regions are created under /sys/class/fpga_region/::
350
351	/sys/class/fpga_region/region0
352	/sys/class/fpga_region/region1
353	/sys/class/fpga_region/region2
354	...
355
356Application needs to search each regionX folder, if feature device is found,
357(e.g. "dfl-port.n" or "dfl-fme.m" is found), then it's the base
358fpga region which represents the FPGA device.
359
360Each base region has one FME and two ports (AFUs) as child devices::
361
362	/sys/class/fpga_region/region0/dfl-fme.0
363	/sys/class/fpga_region/region0/dfl-port.0
364	/sys/class/fpga_region/region0/dfl-port.1
365	...
366
367	/sys/class/fpga_region/region3/dfl-fme.1
368	/sys/class/fpga_region/region3/dfl-port.2
369	/sys/class/fpga_region/region3/dfl-port.3
370	...
371
372In general, the FME/AFU sysfs interfaces are named as follows::
373
374	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/
375	/sys/class/fpga_region/<regionX>/<dfl-port.m>/
376
377with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
378ports.
379
380The device nodes used for ioctl() or mmap() can be referenced through::
381
382	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev
383	/sys/class/fpga_region/<regionX>/<dfl-port.n>/dev
384
385
386Performance Counters
387====================
388Performance reporting is one private feature implemented in FME. It could
389supports several independent, system-wide, device counter sets in hardware to
390monitor and count for performance events, including "basic", "cache", "fabric",
391"vtd" and "vtd_sip" counters. Users could use standard perf tool to monitor
392FPGA cache hit/miss rate, transaction number, interface clock counter of AFU
393and other FPGA performance events.
394
395Different FPGA devices may have different counter sets, depending on hardware
396implementation. E.g., some discrete FPGA cards don't have any cache. User could
397use "perf list" to check which perf events are supported by target hardware.
398
399In order to allow user to use standard perf API to access these performance
400counters, driver creates a perf PMU, and related sysfs interfaces in
401/sys/bus/event_source/devices/dfl_fme* to describe available perf events and
402configuration options.
403
404The "format" directory describes the format of the config field of struct
405perf_event_attr. There are 3 bitfields for config: "evtype" defines which type
406the perf event belongs to; "event" is the identity of the event within its
407category; "portid" is introduced to decide counters set to monitor on FPGA
408overall data or a specific port.
409
410The "events" directory describes the configuration templates for all available
411events which can be used with perf tool directly. For example, fab_mmio_read
412has the configuration "event=0x06,evtype=0x02,portid=0xff", which shows this
413event belongs to fabric type (0x02), the local event id is 0x06 and it is for
414overall monitoring (portid=0xff).
415
416Example usage of perf::
417
418  $# perf list |grep dfl_fme
419
420  dfl_fme0/fab_mmio_read/                              [Kernel PMU event]
421  <...>
422  dfl_fme0/fab_port_mmio_read,portid=?/                [Kernel PMU event]
423  <...>
424
425  $# perf stat -a -e dfl_fme0/fab_mmio_read/ <command>
426  or
427  $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0xff/ <command>
428  or
429  $# perf stat -a -e dfl_fme0/config=0xff2006/ <command>
430
431Another example, fab_port_mmio_read monitors mmio read of a specific port. So
432its configuration template is "event=0x06,evtype=0x01,portid=?". The portid
433should be explicitly set.
434
435Its usage of perf::
436
437  $# perf stat -a -e dfl_fme0/fab_port_mmio_read,portid=0x0/ <command>
438  or
439  $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0x0/ <command>
440  or
441  $# perf stat -a -e dfl_fme0/config=0x2006/ <command>
442
443Please note for fabric counters, overall perf events (fab_*) and port perf
444events (fab_port_*) actually share one set of counters in hardware, so it can't
445monitor both at the same time. If this set of counters is configured to monitor
446overall data, then per port perf data is not supported. See below example::
447
448  $# perf stat -e dfl_fme0/fab_mmio_read/,dfl_fme0/fab_port_mmio_write,\
449                                                    portid=0/ sleep 1
450
451  Performance counter stats for 'system wide':
452
453                 3      dfl_fme0/fab_mmio_read/
454   <not supported>      dfl_fme0/fab_port_mmio_write,portid=0x0/
455
456       1.001750904 seconds time elapsed
457
458The driver also provides a "cpumask" sysfs attribute, which contains only one
459CPU id used to access these perf events. Counting on multiple CPU is not allowed
460since they are system-wide counters on FPGA device.
461
462The current driver does not support sampling. So "perf record" is unsupported.
463
464
465Add new FIUs support
466====================
467It's possible that developers made some new function blocks (FIUs) under this
468DFL framework, then new platform device driver needs to be developed for the
469new feature dev (FIU) following the same way as existing feature dev drivers
470(e.g. FME and Port/AFU platform device driver). Besides that, it requires
471modification on DFL framework enumeration code too, for new FIU type detection
472and related platform devices creation.
473
474
475Add new private features support
476================================
477In some cases, we may need to add some new private features to existing FIUs
478(e.g. FME or Port). Developers don't need to touch enumeration code in DFL
479framework, as each private feature will be parsed automatically and related
480mmio resources can be found under FIU platform device created by DFL framework.
481Developer only needs to provide a sub feature driver with matched feature id.
482FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c)
483could be a reference.
484
485
486Open discussion
487===============
488FME driver exports one ioctl (DFL_FPGA_FME_PORT_PR) for partial reconfiguration
489to user now. In the future, if unified user interfaces for reconfiguration are
490added, FME driver should switch to them from ioctl interface.
491