xref: /openbmc/linux/Documentation/fpga/dfl.rst (revision 547840bd)
1=================================================
2FPGA Device Feature List (DFL) Framework Overview
3=================================================
4
5Authors:
6
7- Enno Luebbers <enno.luebbers@intel.com>
8- Xiao Guangrong <guangrong.xiao@linux.intel.com>
9- Wu Hao <hao.wu@intel.com>
10
11The Device Feature List (DFL) FPGA framework (and drivers according to this
12this framework) hides the very details of low layer hardwares and provides
13unified interfaces to userspace. Applications could use these interfaces to
14configure, enumerate, open and access FPGA accelerators on platforms which
15implement the DFL in the device memory. Besides this, the DFL framework
16enables system level management functions such as FPGA reconfiguration.
17
18
19Device Feature List (DFL) Overview
20==================================
21Device Feature List (DFL) defines a linked list of feature headers within the
22device MMIO space to provide an extensible way of adding features. Software can
23walk through these predefined data structures to enumerate FPGA features:
24FPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features,
25as illustrated below::
26
27    Header            Header            Header            Header
28 +----------+  +-->+----------+  +-->+----------+  +-->+----------+
29 |   Type   |  |   |  Type    |  |   |  Type    |  |   |  Type    |
30 |   FIU    |  |   | Private  |  |   | Private  |  |   | Private  |
31 +----------+  |   | Feature  |  |   | Feature  |  |   | Feature  |
32 | Next_DFH |--+   +----------+  |   +----------+  |   +----------+
33 +----------+      | Next_DFH |--+   | Next_DFH |--+   | Next_DFH |--> NULL
34 |    ID    |      +----------+      +----------+      +----------+
35 +----------+      |    ID    |      |    ID    |      |    ID    |
36 | Next_AFU |--+   +----------+      +----------+      +----------+
37 +----------+  |   | Feature  |      | Feature  |      | Feature  |
38 |  Header  |  |   | Register |      | Register |      | Register |
39 | Register |  |   |   Set    |      |   Set    |      |   Set    |
40 |   Set    |  |   +----------+      +----------+      +----------+
41 +----------+  |      Header
42               +-->+----------+
43                   |   Type   |
44                   |   AFU    |
45                   +----------+
46                   | Next_DFH |--> NULL
47                   +----------+
48                   |   GUID   |
49                   +----------+
50                   |  Header  |
51                   | Register |
52                   |   Set    |
53                   +----------+
54
55FPGA Interface Unit (FIU) represents a standalone functional unit for the
56interface to FPGA, e.g. the FPGA Management Engine (FME) and Port (more
57descriptions on FME and Port in later sections).
58
59Accelerated Function Unit (AFU) represents a FPGA programmable region and
60always connects to a FIU (e.g. a Port) as its child as illustrated above.
61
62Private Features represent sub features of the FIU and AFU. They could be
63various function blocks with different IDs, but all private features which
64belong to the same FIU or AFU, must be linked to one list via the Next Device
65Feature Header (Next_DFH) pointer.
66
67Each FIU, AFU and Private Feature could implement its own functional registers.
68The functional register set for FIU and AFU, is named as Header Register Set,
69e.g. FME Header Register Set, and the one for Private Feature, is named as
70Feature Register Set, e.g. FME Partial Reconfiguration Feature Register Set.
71
72This Device Feature List provides a way of linking features together, it's
73convenient for software to locate each feature by walking through this list,
74and can be implemented in register regions of any FPGA device.
75
76
77FIU - FME (FPGA Management Engine)
78==================================
79The FPGA Management Engine performs reconfiguration and other infrastructure
80functions. Each FPGA device only has one FME.
81
82User-space applications can acquire exclusive access to the FME using open(),
83and release it using close().
84
85The following functions are exposed through ioctls:
86
87- Get driver API version (DFL_FPGA_GET_API_VERSION)
88- Check for extensions (DFL_FPGA_CHECK_EXTENSION)
89- Program bitstream (DFL_FPGA_FME_PORT_PR)
90- Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN)
91- Release port from PF (DFL_FPGA_FME_PORT_RELEASE)
92
93More functions are exposed through sysfs
94(/sys/class/fpga_region/regionX/dfl-fme.n/):
95
96 Read bitstream ID (bitstream_id)
97     bitstream_id indicates version of the static FPGA region.
98
99 Read bitstream metadata (bitstream_metadata)
100     bitstream_metadata includes detailed information of static FPGA region,
101     e.g. synthesis date and seed.
102
103 Read number of ports (ports_num)
104     one FPGA device may have more than one port, this sysfs interface indicates
105     how many ports the FPGA device has.
106
107 Global error reporting management (errors/)
108     error reporting sysfs interfaces allow user to read errors detected by the
109     hardware, and clear the logged errors.
110
111 Power management (dfl_fme_power hwmon)
112     power management hwmon sysfs interfaces allow user to read power management
113     information (power consumption, thresholds, threshold status, limits, etc.)
114     and configure power thresholds for different throttling levels.
115
116 Thermal management (dfl_fme_thermal hwmon)
117     thermal management hwmon sysfs interfaces allow user to read thermal
118     management information (current temperature, thresholds, threshold status,
119     etc.).
120
121
122FIU - PORT
123==========
124A port represents the interface between the static FPGA fabric and a partially
125reconfigurable region containing an AFU. It controls the communication from SW
126to the accelerator and exposes features such as reset and debug. Each FPGA
127device may have more than one port, but always one AFU per port.
128
129
130AFU
131===
132An AFU is attached to a port FIU and exposes a fixed length MMIO region to be
133used for accelerator-specific control registers.
134
135User-space applications can acquire exclusive access to an AFU attached to a
136port by using open() on the port device node and release it using close().
137
138The following functions are exposed through ioctls:
139
140- Get driver API version (DFL_FPGA_GET_API_VERSION)
141- Check for extensions (DFL_FPGA_CHECK_EXTENSION)
142- Get port info (DFL_FPGA_PORT_GET_INFO)
143- Get MMIO region info (DFL_FPGA_PORT_GET_REGION_INFO)
144- Map DMA buffer (DFL_FPGA_PORT_DMA_MAP)
145- Unmap DMA buffer (DFL_FPGA_PORT_DMA_UNMAP)
146- Reset AFU (DFL_FPGA_PORT_RESET)
147
148DFL_FPGA_PORT_RESET:
149  reset the FPGA Port and its AFU. Userspace can do Port
150  reset at any time, e.g. during DMA or Partial Reconfiguration. But it should
151  never cause any system level issue, only functional failure (e.g. DMA or PR
152  operation failure) and be recoverable from the failure.
153
154User-space applications can also mmap() accelerator MMIO regions.
155
156More functions are exposed through sysfs:
157(/sys/class/fpga_region/<regionX>/<dfl-port.m>/):
158
159 Read Accelerator GUID (afu_id)
160     afu_id indicates which PR bitstream is programmed to this AFU.
161
162 Error reporting (errors/)
163     error reporting sysfs interfaces allow user to read port/afu errors
164     detected by the hardware, and clear the logged errors.
165
166
167DFL Framework Overview
168======================
169
170::
171
172         +----------+    +--------+ +--------+ +--------+
173         |   FME    |    |  AFU   | |  AFU   | |  AFU   |
174         |  Module  |    | Module | | Module | | Module |
175         +----------+    +--------+ +--------+ +--------+
176                 +-----------------------+
177                 | FPGA Container Device |    Device Feature List
178                 |  (FPGA Base Region)   |         Framework
179                 +-----------------------+
180  ------------------------------------------------------------------
181               +----------------------------+
182               |   FPGA DFL Device Module   |
183               | (e.g. PCIE/Platform Device)|
184               +----------------------------+
185                 +------------------------+
186                 |  FPGA Hardware Device  |
187                 +------------------------+
188
189DFL framework in kernel provides common interfaces to create container device
190(FPGA base region), discover feature devices and their private features from the
191given Device Feature Lists and create platform devices for feature devices
192(e.g. FME, Port and AFU) with related resources under the container device. It
193also abstracts operations for the private features and exposes common ops to
194feature device drivers.
195
196The FPGA DFL Device could be different hardwares, e.g. PCIe device, platform
197device and etc. Its driver module is always loaded first once the device is
198created by the system. This driver plays an infrastructural role in the
199driver architecture. It locates the DFLs in the device memory, handles them
200and related resources to common interfaces from DFL framework for enumeration.
201(Please refer to drivers/fpga/dfl.c for detailed enumeration APIs).
202
203The FPGA Management Engine (FME) driver is a platform driver which is loaded
204automatically after FME platform device creation from the DFL device module. It
205provides the key features for FPGA management, including:
206
207	a) Expose static FPGA region information, e.g. version and metadata.
208	   Users can read related information via sysfs interfaces exposed
209	   by FME driver.
210
211	b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
212	   bridges and FPGA regions during PR sub feature initialization. Once
213	   it receives a DFL_FPGA_FME_PORT_PR ioctl from user, it invokes the
214	   common interface function from FPGA Region to complete the partial
215	   reconfiguration of the PR bitstream to the given port.
216
217Similar to the FME driver, the FPGA Accelerated Function Unit (AFU) driver is
218probed once the AFU platform device is created. The main function of this module
219is to provide an interface for userspace applications to access the individual
220accelerators, including basic reset control on port, AFU MMIO region export, dma
221buffer mapping service functions.
222
223After feature platform devices creation, matched platform drivers will be loaded
224automatically to handle different functionalities. Please refer to next sections
225for detailed information on functional units which have been already implemented
226under this DFL framework.
227
228
229Partial Reconfiguration
230=======================
231As mentioned above, accelerators can be reconfigured through partial
232reconfiguration of a PR bitstream file. The PR bitstream file must have been
233generated for the exact static FPGA region and targeted reconfigurable region
234(port) of the FPGA, otherwise, the reconfiguration operation will fail and
235possibly cause system instability. This compatibility can be checked by
236comparing the compatibility ID noted in the header of PR bitstream file against
237the compat_id exposed by the target FPGA region. This check is usually done by
238userspace before calling the reconfiguration IOCTL.
239
240
241FPGA virtualization - PCIe SRIOV
242================================
243This section describes the virtualization support on DFL based FPGA device to
244enable accessing an accelerator from applications running in a virtual machine
245(VM). This section only describes the PCIe based FPGA device with SRIOV support.
246
247Features supported by the particular FPGA device are exposed through Device
248Feature Lists, as illustrated below:
249
250::
251
252    +-------------------------------+  +-------------+
253    |              PF               |  |     VF      |
254    +-------------------------------+  +-------------+
255        ^            ^         ^              ^
256        |            |         |              |
257  +-----|------------|---------|--------------|-------+
258  |     |            |         |              |       |
259  |  +-----+     +-------+ +-------+      +-------+   |
260  |  | FME |     | Port0 | | Port1 |      | Port2 |   |
261  |  +-----+     +-------+ +-------+      +-------+   |
262  |                  ^         ^              ^       |
263  |                  |         |              |       |
264  |              +-------+ +------+       +-------+   |
265  |              |  AFU  | |  AFU |       |  AFU  |   |
266  |              +-------+ +------+       +-------+   |
267  |                                                   |
268  |            DFL based FPGA PCIe Device             |
269  +---------------------------------------------------+
270
271FME is always accessed through the physical function (PF).
272
273Ports (and related AFUs) are accessed via PF by default, but could be exposed
274through virtual function (VF) devices via PCIe SRIOV. Each VF only contains
2751 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)
276created via PCIe SRIOV interface, to virtual machines.
277
278The driver organization in virtualization case is illustrated below:
279::
280
281    +-------++------++------+             |
282    | FME   || FME  || FME  |             |
283    | FPGA  || FPGA || FPGA |             |
284    |Manager||Bridge||Region|             |
285    +-------++------++------+             |
286    +-----------------------+  +--------+ |             +--------+
287    |          FME          |  |  AFU   | |             |  AFU   |
288    |         Module        |  | Module | |             | Module |
289    +-----------------------+  +--------+ |             +--------+
290          +-----------------------+       |       +-----------------------+
291          | FPGA Container Device |       |       | FPGA Container Device |
292          |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |
293          +-----------------------+       |       +-----------------------+
294            +------------------+          |         +------------------+
295            | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
296            +------------------+   Host   | Machine +------------------+
297   -------------------------------------- | ------------------------------
298             +---------------+            |          +---------------+
299             | PCI PF Device |            |          | PCI VF Device |
300             +---------------+            |          +---------------+
301
302FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device
303is detected. It:
304
305* Finishes enumeration on both FPGA PCIe PF and VF device using common
306  interfaces from DFL framework.
307* Supports SRIOV.
308
309The FME device driver plays a management role in this driver architecture, it
310provides ioctls to release Port from PF and assign Port to PF. After release
311a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV
312sysfs interface.
313
314To enable accessing an accelerator from applications running in a VM, the
315respective AFU's port needs to be assigned to a VF using the following steps:
316
317#. The PF owns all AFU ports by default. Any port that needs to be
318   reassigned to a VF must first be released through the
319   DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device.
320
321#. Once N ports are released from PF, then user can use command below
322   to enable SRIOV and VFs. Each VF owns only one Port with AFU.
323
324   ::
325
326      echo N > $PCI_DEVICE_PATH/sriov_numvfs
327
328#. Pass through the VFs to VMs
329
330#. The AFU under VF is accessible from applications in VM (using the
331   same driver inside the VF).
332
333Note that an FME can't be assigned to a VF, thus PR and other management
334functions are only available via the PF.
335
336Device enumeration
337==================
338This section introduces how applications enumerate the fpga device from
339the sysfs hierarchy under /sys/class/fpga_region.
340
341In the example below, two DFL based FPGA devices are installed in the host. Each
342fpga device has one FME and two ports (AFUs).
343
344FPGA regions are created under /sys/class/fpga_region/::
345
346	/sys/class/fpga_region/region0
347	/sys/class/fpga_region/region1
348	/sys/class/fpga_region/region2
349	...
350
351Application needs to search each regionX folder, if feature device is found,
352(e.g. "dfl-port.n" or "dfl-fme.m" is found), then it's the base
353fpga region which represents the FPGA device.
354
355Each base region has one FME and two ports (AFUs) as child devices::
356
357	/sys/class/fpga_region/region0/dfl-fme.0
358	/sys/class/fpga_region/region0/dfl-port.0
359	/sys/class/fpga_region/region0/dfl-port.1
360	...
361
362	/sys/class/fpga_region/region3/dfl-fme.1
363	/sys/class/fpga_region/region3/dfl-port.2
364	/sys/class/fpga_region/region3/dfl-port.3
365	...
366
367In general, the FME/AFU sysfs interfaces are named as follows::
368
369	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/
370	/sys/class/fpga_region/<regionX>/<dfl-port.m>/
371
372with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
373ports.
374
375The device nodes used for ioctl() or mmap() can be referenced through::
376
377	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev
378	/sys/class/fpga_region/<regionX>/<dfl-port.n>/dev
379
380
381Add new FIUs support
382====================
383It's possible that developers made some new function blocks (FIUs) under this
384DFL framework, then new platform device driver needs to be developed for the
385new feature dev (FIU) following the same way as existing feature dev drivers
386(e.g. FME and Port/AFU platform device driver). Besides that, it requires
387modification on DFL framework enumeration code too, for new FIU type detection
388and related platform devices creation.
389
390
391Add new private features support
392================================
393In some cases, we may need to add some new private features to existing FIUs
394(e.g. FME or Port). Developers don't need to touch enumeration code in DFL
395framework, as each private feature will be parsed automatically and related
396mmio resources can be found under FIU platform device created by DFL framework.
397Developer only needs to provide a sub feature driver with matched feature id.
398FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c)
399could be a reference.
400
401
402Open discussion
403===============
404FME driver exports one ioctl (DFL_FPGA_FME_PORT_PR) for partial reconfiguration
405to user now. In the future, if unified user interfaces for reconfiguration are
406added, FME driver should switch to them from ioctl interface.
407