xref: /openbmc/linux/Documentation/fpga/dfl.rst (revision b830f94f)
1=================================================
2FPGA Device Feature List (DFL) Framework Overview
3=================================================
4
5Authors:
6
7- Enno Luebbers <enno.luebbers@intel.com>
8- Xiao Guangrong <guangrong.xiao@linux.intel.com>
9- Wu Hao <hao.wu@intel.com>
10
11The Device Feature List (DFL) FPGA framework (and drivers according to this
12this framework) hides the very details of low layer hardwares and provides
13unified interfaces to userspace. Applications could use these interfaces to
14configure, enumerate, open and access FPGA accelerators on platforms which
15implement the DFL in the device memory. Besides this, the DFL framework
16enables system level management functions such as FPGA reconfiguration.
17
18
19Device Feature List (DFL) Overview
20==================================
21Device Feature List (DFL) defines a linked list of feature headers within the
22device MMIO space to provide an extensible way of adding features. Software can
23walk through these predefined data structures to enumerate FPGA features:
24FPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features,
25as illustrated below::
26
27    Header            Header            Header            Header
28 +----------+  +-->+----------+  +-->+----------+  +-->+----------+
29 |   Type   |  |   |  Type    |  |   |  Type    |  |   |  Type    |
30 |   FIU    |  |   | Private  |  |   | Private  |  |   | Private  |
31 +----------+  |   | Feature  |  |   | Feature  |  |   | Feature  |
32 | Next_DFH |--+   +----------+  |   +----------+  |   +----------+
33 +----------+      | Next_DFH |--+   | Next_DFH |--+   | Next_DFH |--> NULL
34 |    ID    |      +----------+      +----------+      +----------+
35 +----------+      |    ID    |      |    ID    |      |    ID    |
36 | Next_AFU |--+   +----------+      +----------+      +----------+
37 +----------+  |   | Feature  |      | Feature  |      | Feature  |
38 |  Header  |  |   | Register |      | Register |      | Register |
39 | Register |  |   |   Set    |      |   Set    |      |   Set    |
40 |   Set    |  |   +----------+      +----------+      +----------+
41 +----------+  |      Header
42               +-->+----------+
43                   |   Type   |
44                   |   AFU    |
45                   +----------+
46                   | Next_DFH |--> NULL
47                   +----------+
48                   |   GUID   |
49                   +----------+
50                   |  Header  |
51                   | Register |
52                   |   Set    |
53                   +----------+
54
55FPGA Interface Unit (FIU) represents a standalone functional unit for the
56interface to FPGA, e.g. the FPGA Management Engine (FME) and Port (more
57descriptions on FME and Port in later sections).
58
59Accelerated Function Unit (AFU) represents a FPGA programmable region and
60always connects to a FIU (e.g. a Port) as its child as illustrated above.
61
62Private Features represent sub features of the FIU and AFU. They could be
63various function blocks with different IDs, but all private features which
64belong to the same FIU or AFU, must be linked to one list via the Next Device
65Feature Header (Next_DFH) pointer.
66
67Each FIU, AFU and Private Feature could implement its own functional registers.
68The functional register set for FIU and AFU, is named as Header Register Set,
69e.g. FME Header Register Set, and the one for Private Feature, is named as
70Feature Register Set, e.g. FME Partial Reconfiguration Feature Register Set.
71
72This Device Feature List provides a way of linking features together, it's
73convenient for software to locate each feature by walking through this list,
74and can be implemented in register regions of any FPGA device.
75
76
77FIU - FME (FPGA Management Engine)
78==================================
79The FPGA Management Engine performs reconfiguration and other infrastructure
80functions. Each FPGA device only has one FME.
81
82User-space applications can acquire exclusive access to the FME using open(),
83and release it using close().
84
85The following functions are exposed through ioctls:
86
87- Get driver API version (DFL_FPGA_GET_API_VERSION)
88- Check for extensions (DFL_FPGA_CHECK_EXTENSION)
89- Program bitstream (DFL_FPGA_FME_PORT_PR)
90
91More functions are exposed through sysfs
92(/sys/class/fpga_region/regionX/dfl-fme.n/):
93
94 Read bitstream ID (bitstream_id)
95     bitstream_id indicates version of the static FPGA region.
96
97 Read bitstream metadata (bitstream_metadata)
98     bitstream_metadata includes detailed information of static FPGA region,
99     e.g. synthesis date and seed.
100
101 Read number of ports (ports_num)
102     one FPGA device may have more than one port, this sysfs interface indicates
103     how many ports the FPGA device has.
104
105
106FIU - PORT
107==========
108A port represents the interface between the static FPGA fabric and a partially
109reconfigurable region containing an AFU. It controls the communication from SW
110to the accelerator and exposes features such as reset and debug. Each FPGA
111device may have more than one port, but always one AFU per port.
112
113
114AFU
115===
116An AFU is attached to a port FIU and exposes a fixed length MMIO region to be
117used for accelerator-specific control registers.
118
119User-space applications can acquire exclusive access to an AFU attached to a
120port by using open() on the port device node and release it using close().
121
122The following functions are exposed through ioctls:
123
124- Get driver API version (DFL_FPGA_GET_API_VERSION)
125- Check for extensions (DFL_FPGA_CHECK_EXTENSION)
126- Get port info (DFL_FPGA_PORT_GET_INFO)
127- Get MMIO region info (DFL_FPGA_PORT_GET_REGION_INFO)
128- Map DMA buffer (DFL_FPGA_PORT_DMA_MAP)
129- Unmap DMA buffer (DFL_FPGA_PORT_DMA_UNMAP)
130- Reset AFU (DFL_FPGA_PORT_RESET)
131
132DFL_FPGA_PORT_RESET:
133  reset the FPGA Port and its AFU. Userspace can do Port
134  reset at any time, e.g. during DMA or Partial Reconfiguration. But it should
135  never cause any system level issue, only functional failure (e.g. DMA or PR
136  operation failure) and be recoverable from the failure.
137
138User-space applications can also mmap() accelerator MMIO regions.
139
140More functions are exposed through sysfs:
141(/sys/class/fpga_region/<regionX>/<dfl-port.m>/):
142
143 Read Accelerator GUID (afu_id)
144     afu_id indicates which PR bitstream is programmed to this AFU.
145
146
147DFL Framework Overview
148======================
149
150::
151
152         +----------+    +--------+ +--------+ +--------+
153         |   FME    |    |  AFU   | |  AFU   | |  AFU   |
154         |  Module  |    | Module | | Module | | Module |
155         +----------+    +--------+ +--------+ +--------+
156                 +-----------------------+
157                 | FPGA Container Device |    Device Feature List
158                 |  (FPGA Base Region)   |         Framework
159                 +-----------------------+
160  ------------------------------------------------------------------
161               +----------------------------+
162               |   FPGA DFL Device Module   |
163               | (e.g. PCIE/Platform Device)|
164               +----------------------------+
165                 +------------------------+
166                 |  FPGA Hardware Device  |
167                 +------------------------+
168
169DFL framework in kernel provides common interfaces to create container device
170(FPGA base region), discover feature devices and their private features from the
171given Device Feature Lists and create platform devices for feature devices
172(e.g. FME, Port and AFU) with related resources under the container device. It
173also abstracts operations for the private features and exposes common ops to
174feature device drivers.
175
176The FPGA DFL Device could be different hardwares, e.g. PCIe device, platform
177device and etc. Its driver module is always loaded first once the device is
178created by the system. This driver plays an infrastructural role in the
179driver architecture. It locates the DFLs in the device memory, handles them
180and related resources to common interfaces from DFL framework for enumeration.
181(Please refer to drivers/fpga/dfl.c for detailed enumeration APIs).
182
183The FPGA Management Engine (FME) driver is a platform driver which is loaded
184automatically after FME platform device creation from the DFL device module. It
185provides the key features for FPGA management, including:
186
187	a) Expose static FPGA region information, e.g. version and metadata.
188	   Users can read related information via sysfs interfaces exposed
189	   by FME driver.
190
191	b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
192	   bridges and FPGA regions during PR sub feature initialization. Once
193	   it receives a DFL_FPGA_FME_PORT_PR ioctl from user, it invokes the
194	   common interface function from FPGA Region to complete the partial
195	   reconfiguration of the PR bitstream to the given port.
196
197Similar to the FME driver, the FPGA Accelerated Function Unit (AFU) driver is
198probed once the AFU platform device is created. The main function of this module
199is to provide an interface for userspace applications to access the individual
200accelerators, including basic reset control on port, AFU MMIO region export, dma
201buffer mapping service functions.
202
203After feature platform devices creation, matched platform drivers will be loaded
204automatically to handle different functionalities. Please refer to next sections
205for detailed information on functional units which have been already implemented
206under this DFL framework.
207
208
209Partial Reconfiguration
210=======================
211As mentioned above, accelerators can be reconfigured through partial
212reconfiguration of a PR bitstream file. The PR bitstream file must have been
213generated for the exact static FPGA region and targeted reconfigurable region
214(port) of the FPGA, otherwise, the reconfiguration operation will fail and
215possibly cause system instability. This compatibility can be checked by
216comparing the compatibility ID noted in the header of PR bitstream file against
217the compat_id exposed by the target FPGA region. This check is usually done by
218userspace before calling the reconfiguration IOCTL.
219
220
221Device enumeration
222==================
223This section introduces how applications enumerate the fpga device from
224the sysfs hierarchy under /sys/class/fpga_region.
225
226In the example below, two DFL based FPGA devices are installed in the host. Each
227fpga device has one FME and two ports (AFUs).
228
229FPGA regions are created under /sys/class/fpga_region/::
230
231	/sys/class/fpga_region/region0
232	/sys/class/fpga_region/region1
233	/sys/class/fpga_region/region2
234	...
235
236Application needs to search each regionX folder, if feature device is found,
237(e.g. "dfl-port.n" or "dfl-fme.m" is found), then it's the base
238fpga region which represents the FPGA device.
239
240Each base region has one FME and two ports (AFUs) as child devices::
241
242	/sys/class/fpga_region/region0/dfl-fme.0
243	/sys/class/fpga_region/region0/dfl-port.0
244	/sys/class/fpga_region/region0/dfl-port.1
245	...
246
247	/sys/class/fpga_region/region3/dfl-fme.1
248	/sys/class/fpga_region/region3/dfl-port.2
249	/sys/class/fpga_region/region3/dfl-port.3
250	...
251
252In general, the FME/AFU sysfs interfaces are named as follows::
253
254	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/
255	/sys/class/fpga_region/<regionX>/<dfl-port.m>/
256
257with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
258ports.
259
260The device nodes used for ioctl() or mmap() can be referenced through::
261
262	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev
263	/sys/class/fpga_region/<regionX>/<dfl-port.n>/dev
264
265
266Add new FIUs support
267====================
268It's possible that developers made some new function blocks (FIUs) under this
269DFL framework, then new platform device driver needs to be developed for the
270new feature dev (FIU) following the same way as existing feature dev drivers
271(e.g. FME and Port/AFU platform device driver). Besides that, it requires
272modification on DFL framework enumeration code too, for new FIU type detection
273and related platform devices creation.
274
275
276Add new private features support
277================================
278In some cases, we may need to add some new private features to existing FIUs
279(e.g. FME or Port). Developers don't need to touch enumeration code in DFL
280framework, as each private feature will be parsed automatically and related
281mmio resources can be found under FIU platform device created by DFL framework.
282Developer only needs to provide a sub feature driver with matched feature id.
283FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c)
284could be a reference.
285
286
287Open discussion
288===============
289FME driver exports one ioctl (DFL_FPGA_FME_PORT_PR) for partial reconfiguration
290to user now. In the future, if unified user interfaces for reconfiguration are
291added, FME driver should switch to them from ioctl interface.
292