xref: /openbmc/linux/Documentation/fpga/dfl.rst (revision 2d99a7ec) 
1=================================================
2FPGA Device Feature List (DFL) Framework Overview
3=================================================
4
5Authors:
6
7- Enno Luebbers <enno.luebbers@intel.com>
8- Xiao Guangrong <guangrong.xiao@linux.intel.com>
9- Wu Hao <hao.wu@intel.com>
10- Xu Yilun <yilun.xu@intel.com>
11
12The Device Feature List (DFL) FPGA framework (and drivers according to
13this framework) hides the very details of low layer hardware and provides
14unified interfaces to userspace. Applications could use these interfaces to
15configure, enumerate, open and access FPGA accelerators on platforms which
16implement the DFL in the device memory. Besides this, the DFL framework
17enables system level management functions such as FPGA reconfiguration.
18
19
20Device Feature List (DFL) Overview
21==================================
22Device Feature List (DFL) defines a linked list of feature headers within the
23device MMIO space to provide an extensible way of adding features. Software can
24walk through these predefined data structures to enumerate FPGA features:
25FPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features,
26as illustrated below::
27
28    Header            Header            Header            Header
29 +----------+  +-->+----------+  +-->+----------+  +-->+----------+
30 |   Type   |  |   |  Type    |  |   |  Type    |  |   |  Type    |
31 |   FIU    |  |   | Private  |  |   | Private  |  |   | Private  |
32 +----------+  |   | Feature  |  |   | Feature  |  |   | Feature  |
33 | Next_DFH |--+   +----------+  |   +----------+  |   +----------+
34 +----------+      | Next_DFH |--+   | Next_DFH |--+   | Next_DFH |--> NULL
35 |    ID    |      +----------+      +----------+      +----------+
36 +----------+      |    ID    |      |    ID    |      |    ID    |
37 | Next_AFU |--+   +----------+      +----------+      +----------+
38 +----------+  |   | Feature  |      | Feature  |      | Feature  |
39 |  Header  |  |   | Register |      | Register |      | Register |
40 | Register |  |   |   Set    |      |   Set    |      |   Set    |
41 |   Set    |  |   +----------+      +----------+      +----------+
42 +----------+  |      Header
43               +-->+----------+
44                   |   Type   |
45                   |   AFU    |
46                   +----------+
47                   | Next_DFH |--> NULL
48                   +----------+
49                   |   GUID   |
50                   +----------+
51                   |  Header  |
52                   | Register |
53                   |   Set    |
54                   +----------+
55
56FPGA Interface Unit (FIU) represents a standalone functional unit for the
57interface to FPGA, e.g. the FPGA Management Engine (FME) and Port (more
58descriptions on FME and Port in later sections).
59
60Accelerated Function Unit (AFU) represents an FPGA programmable region and
61always connects to a FIU (e.g. a Port) as its child as illustrated above.
62
63Private Features represent sub features of the FIU and AFU. They could be
64various function blocks with different IDs, but all private features which
65belong to the same FIU or AFU, must be linked to one list via the Next Device
66Feature Header (Next_DFH) pointer.
67
68Each FIU, AFU and Private Feature could implement its own functional registers.
69The functional register set for FIU and AFU, is named as Header Register Set,
70e.g. FME Header Register Set, and the one for Private Feature, is named as
71Feature Register Set, e.g. FME Partial Reconfiguration Feature Register Set.
72
73This Device Feature List provides a way of linking features together, it's
74convenient for software to locate each feature by walking through this list,
75and can be implemented in register regions of any FPGA device.
76
77
78Device Feature Header - Version 0
79=================================
80Version 0 (DFHv0) is the original version of the Device Feature Header.
81All multi-byte quantities in DFHv0 are little-endian.
82The format of DFHv0 is shown below::
83
84    +-----------------------------------------------------------------------+
85    |63 Type 60|59 DFH VER 52|51 Rsvd 41|40 EOL|39 Next 16|15 REV 12|11 ID 0| 0x00
86    +-----------------------------------------------------------------------+
87    |63                                 GUID_L                             0| 0x08
88    +-----------------------------------------------------------------------+
89    |63                                 GUID_H                             0| 0x10
90    +-----------------------------------------------------------------------+
91
92- Offset 0x00
93
94  * Type - The type of DFH (e.g. FME, AFU, or private feature).
95  * DFH VER - The version of the DFH.
96  * Rsvd - Currently unused.
97  * EOL - Set if the DFH is the end of the Device Feature List (DFL).
98  * Next - The offset in bytes of the next DFH in the DFL from the DFH start,
99    and the start of a DFH must be aligned to an 8 byte boundary.
100    If EOL is set, Next is the size of MMIO of the last feature in the list.
101  * REV - The revision of the feature associated with this header.
102  * ID - The feature ID if Type is private feature.
103
104- Offset 0x08
105
106  * GUID_L - Least significant 64 bits of a 128-bit Globally Unique Identifier
107    (present only if Type is FME or AFU).
108
109- Offset 0x10
110
111  * GUID_H - Most significant 64 bits of a 128-bit Globally Unique Identifier
112    (present only if Type is FME or AFU).
113
114
115Device Feature Header - Version 1
116=================================
117Version 1 (DFHv1) of the Device Feature Header adds the following functionality:
118
119* Provides a standardized mechanism for features to describe
120  parameters/capabilities to software.
121* Standardize the use of a GUID for all DFHv1 types.
122* Decouples the DFH location from the register space of the feature itself.
123
124All multi-byte quantities in DFHv1 are little-endian.
125The format of Version 1 of the Device Feature Header (DFH) is shown below::
126
127    +-----------------------------------------------------------------------+
128    |63 Type 60|59 DFH VER 52|51 Rsvd 41|40 EOL|39 Next 16|15 REV 12|11 ID 0| 0x00
129    +-----------------------------------------------------------------------+
130    |63                                 GUID_L                             0| 0x08
131    +-----------------------------------------------------------------------+
132    |63                                 GUID_H                             0| 0x10
133    +-----------------------------------------------------------------------+
134    |63                   Reg Address/Offset                      1|  Rel  0| 0x18
135    +-----------------------------------------------------------------------+
136    |63        Reg Size       32|Params 31|30 Group    16|15 Instance      0| 0x20
137    +-----------------------------------------------------------------------+
138    |63 Next    35|34RSV33|EOP32|31 Param Version 16|15 Param ID           0| 0x28
139    +-----------------------------------------------------------------------+
140    |63                 Parameter Data                                     0| 0x30
141    +-----------------------------------------------------------------------+
142
143                                  ...
144
145    +-----------------------------------------------------------------------+
146    |63 Next    35|34RSV33|EOP32|31 Param Version 16|15 Param ID           0|
147    +-----------------------------------------------------------------------+
148    |63                 Parameter Data                                     0|
149    +-----------------------------------------------------------------------+
150
151- Offset 0x00
152
153  * Type - The type of DFH (e.g. FME, AFU, or private feature).
154  * DFH VER - The version of the DFH.
155  * Rsvd - Currently unused.
156  * EOL - Set if the DFH is the end of the Device Feature List (DFL).
157  * Next - The offset in bytes of the next DFH in the DFL from the DFH start,
158    and the start of a DFH must be aligned to an 8 byte boundary.
159    If EOL is set, Next is the size of MMIO of the last feature in the list.
160  * REV - The revision of the feature associated with this header.
161  * ID - The feature ID if Type is private feature.
162
163- Offset 0x08
164
165  * GUID_L - Least significant 64 bits of a 128-bit Globally Unique Identifier.
166
167- Offset 0x10
168
169  * GUID_H - Most significant 64 bits of a 128-bit Globally Unique Identifier.
170
171- Offset 0x18
172
173  * Reg Address/Offset - If Rel bit is set, then the value is the high 63 bits
174    of a 16-bit aligned absolute address of the feature's registers. Otherwise
175    the value is the offset from the start of the DFH of the feature's registers.
176
177- Offset 0x20
178
179  * Reg Size - Size of feature's register set in bytes.
180  * Params - Set if DFH has a list of parameter blocks.
181  * Group - Id of group if feature is part of a group.
182  * Instance - Id of feature instance within a group.
183
184- Offset 0x28 if feature has parameters
185
186  * Next - Offset to the next parameter block in 8 byte words. If EOP set,
187    size in 8 byte words of last parameter.
188  * Param Version - Version of Param ID.
189  * Param ID - ID of parameter.
190
191- Offset 0x30
192
193  * Parameter Data - Parameter data whose size and format is defined by
194    version and ID of the parameter.
195
196
197FIU - FME (FPGA Management Engine)
198==================================
199The FPGA Management Engine performs reconfiguration and other infrastructure
200functions. Each FPGA device only has one FME.
201
202User-space applications can acquire exclusive access to the FME using open(),
203and release it using close().
204
205The following functions are exposed through ioctls:
206
207- Get driver API version (DFL_FPGA_GET_API_VERSION)
208- Check for extensions (DFL_FPGA_CHECK_EXTENSION)
209- Program bitstream (DFL_FPGA_FME_PORT_PR)
210- Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN)
211- Release port from PF (DFL_FPGA_FME_PORT_RELEASE)
212- Get number of irqs of FME global error (DFL_FPGA_FME_ERR_GET_IRQ_NUM)
213- Set interrupt trigger for FME error (DFL_FPGA_FME_ERR_SET_IRQ)
214
215More functions are exposed through sysfs
216(/sys/class/fpga_region/regionX/dfl-fme.n/):
217
218 Read bitstream ID (bitstream_id)
219     bitstream_id indicates version of the static FPGA region.
220
221 Read bitstream metadata (bitstream_metadata)
222     bitstream_metadata includes detailed information of static FPGA region,
223     e.g. synthesis date and seed.
224
225 Read number of ports (ports_num)
226     one FPGA device may have more than one port, this sysfs interface indicates
227     how many ports the FPGA device has.
228
229 Global error reporting management (errors/)
230     error reporting sysfs interfaces allow user to read errors detected by the
231     hardware, and clear the logged errors.
232
233 Power management (dfl_fme_power hwmon)
234     power management hwmon sysfs interfaces allow user to read power management
235     information (power consumption, thresholds, threshold status, limits, etc.)
236     and configure power thresholds for different throttling levels.
237
238 Thermal management (dfl_fme_thermal hwmon)
239     thermal management hwmon sysfs interfaces allow user to read thermal
240     management information (current temperature, thresholds, threshold status,
241     etc.).
242
243 Performance reporting
244     performance counters are exposed through perf PMU APIs. Standard perf tool
245     can be used to monitor all available perf events. Please see performance
246     counter section below for more detailed information.
247
248
249FIU - PORT
250==========
251A port represents the interface between the static FPGA fabric and a partially
252reconfigurable region containing an AFU. It controls the communication from SW
253to the accelerator and exposes features such as reset and debug. Each FPGA
254device may have more than one port, but always one AFU per port.
255
256
257AFU
258===
259An AFU is attached to a port FIU and exposes a fixed length MMIO region to be
260used for accelerator-specific control registers.
261
262User-space applications can acquire exclusive access to an AFU attached to a
263port by using open() on the port device node and release it using close().
264
265The following functions are exposed through ioctls:
266
267- Get driver API version (DFL_FPGA_GET_API_VERSION)
268- Check for extensions (DFL_FPGA_CHECK_EXTENSION)
269- Get port info (DFL_FPGA_PORT_GET_INFO)
270- Get MMIO region info (DFL_FPGA_PORT_GET_REGION_INFO)
271- Map DMA buffer (DFL_FPGA_PORT_DMA_MAP)
272- Unmap DMA buffer (DFL_FPGA_PORT_DMA_UNMAP)
273- Reset AFU (DFL_FPGA_PORT_RESET)
274- Get number of irqs of port error (DFL_FPGA_PORT_ERR_GET_IRQ_NUM)
275- Set interrupt trigger for port error (DFL_FPGA_PORT_ERR_SET_IRQ)
276- Get number of irqs of UINT (DFL_FPGA_PORT_UINT_GET_IRQ_NUM)
277- Set interrupt trigger for UINT (DFL_FPGA_PORT_UINT_SET_IRQ)
278
279DFL_FPGA_PORT_RESET:
280  reset the FPGA Port and its AFU. Userspace can do Port
281  reset at any time, e.g. during DMA or Partial Reconfiguration. But it should
282  never cause any system level issue, only functional failure (e.g. DMA or PR
283  operation failure) and be recoverable from the failure.
284
285User-space applications can also mmap() accelerator MMIO regions.
286
287More functions are exposed through sysfs:
288(/sys/class/fpga_region/<regionX>/<dfl-port.m>/):
289
290 Read Accelerator GUID (afu_id)
291     afu_id indicates which PR bitstream is programmed to this AFU.
292
293 Error reporting (errors/)
294     error reporting sysfs interfaces allow user to read port/afu errors
295     detected by the hardware, and clear the logged errors.
296
297
298DFL Framework Overview
299======================
300
301::
302
303         +----------+    +--------+ +--------+ +--------+
304         |   FME    |    |  AFU   | |  AFU   | |  AFU   |
305         |  Module  |    | Module | | Module | | Module |
306         +----------+    +--------+ +--------+ +--------+
307                 +-----------------------+
308                 | FPGA Container Device |    Device Feature List
309                 |  (FPGA Base Region)   |         Framework
310                 +-----------------------+
311  ------------------------------------------------------------------
312               +----------------------------+
313               |   FPGA DFL Device Module   |
314               | (e.g. PCIE/Platform Device)|
315               +----------------------------+
316                 +------------------------+
317                 |  FPGA Hardware Device  |
318                 +------------------------+
319
320DFL framework in kernel provides common interfaces to create container device
321(FPGA base region), discover feature devices and their private features from the
322given Device Feature Lists and create platform devices for feature devices
323(e.g. FME, Port and AFU) with related resources under the container device. It
324also abstracts operations for the private features and exposes common ops to
325feature device drivers.
326
327The FPGA DFL Device could be different hardware, e.g. PCIe device, platform
328device and etc. Its driver module is always loaded first once the device is
329created by the system. This driver plays an infrastructural role in the
330driver architecture. It locates the DFLs in the device memory, handles them
331and related resources to common interfaces from DFL framework for enumeration.
332(Please refer to drivers/fpga/dfl.c for detailed enumeration APIs).
333
334The FPGA Management Engine (FME) driver is a platform driver which is loaded
335automatically after FME platform device creation from the DFL device module. It
336provides the key features for FPGA management, including:
337
338	a) Expose static FPGA region information, e.g. version and metadata.
339	   Users can read related information via sysfs interfaces exposed
340	   by FME driver.
341
342	b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
343	   bridges and FPGA regions during PR sub feature initialization. Once
344	   it receives a DFL_FPGA_FME_PORT_PR ioctl from user, it invokes the
345	   common interface function from FPGA Region to complete the partial
346	   reconfiguration of the PR bitstream to the given port.
347
348Similar to the FME driver, the FPGA Accelerated Function Unit (AFU) driver is
349probed once the AFU platform device is created. The main function of this module
350is to provide an interface for userspace applications to access the individual
351accelerators, including basic reset control on port, AFU MMIO region export, dma
352buffer mapping service functions.
353
354After feature platform devices creation, matched platform drivers will be loaded
355automatically to handle different functionalities. Please refer to next sections
356for detailed information on functional units which have been already implemented
357under this DFL framework.
358
359
360Partial Reconfiguration
361=======================
362As mentioned above, accelerators can be reconfigured through partial
363reconfiguration of a PR bitstream file. The PR bitstream file must have been
364generated for the exact static FPGA region and targeted reconfigurable region
365(port) of the FPGA, otherwise, the reconfiguration operation will fail and
366possibly cause system instability. This compatibility can be checked by
367comparing the compatibility ID noted in the header of PR bitstream file against
368the compat_id exposed by the target FPGA region. This check is usually done by
369userspace before calling the reconfiguration IOCTL.
370
371
372FPGA virtualization - PCIe SRIOV
373================================
374This section describes the virtualization support on DFL based FPGA device to
375enable accessing an accelerator from applications running in a virtual machine
376(VM). This section only describes the PCIe based FPGA device with SRIOV support.
377
378Features supported by the particular FPGA device are exposed through Device
379Feature Lists, as illustrated below:
380
381::
382
383    +-------------------------------+  +-------------+
384    |              PF               |  |     VF      |
385    +-------------------------------+  +-------------+
386        ^            ^         ^              ^
387        |            |         |              |
388  +-----|------------|---------|--------------|-------+
389  |     |            |         |              |       |
390  |  +-----+     +-------+ +-------+      +-------+   |
391  |  | FME |     | Port0 | | Port1 |      | Port2 |   |
392  |  +-----+     +-------+ +-------+      +-------+   |
393  |                  ^         ^              ^       |
394  |                  |         |              |       |
395  |              +-------+ +------+       +-------+   |
396  |              |  AFU  | |  AFU |       |  AFU  |   |
397  |              +-------+ +------+       +-------+   |
398  |                                                   |
399  |            DFL based FPGA PCIe Device             |
400  +---------------------------------------------------+
401
402FME is always accessed through the physical function (PF).
403
404Ports (and related AFUs) are accessed via PF by default, but could be exposed
405through virtual function (VF) devices via PCIe SRIOV. Each VF only contains
4061 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)
407created via PCIe SRIOV interface, to virtual machines.
408
409The driver organization in virtualization case is illustrated below:
410::
411
412    +-------++------++------+             |
413    | FME   || FME  || FME  |             |
414    | FPGA  || FPGA || FPGA |             |
415    |Manager||Bridge||Region|             |
416    +-------++------++------+             |
417    +-----------------------+  +--------+ |             +--------+
418    |          FME          |  |  AFU   | |             |  AFU   |
419    |         Module        |  | Module | |             | Module |
420    +-----------------------+  +--------+ |             +--------+
421          +-----------------------+       |       +-----------------------+
422          | FPGA Container Device |       |       | FPGA Container Device |
423          |  (FPGA Base Region)   |       |       |  (FPGA Base Region)   |
424          +-----------------------+       |       +-----------------------+
425            +------------------+          |         +------------------+
426            | FPGA PCIE Module |          | Virtual | FPGA PCIE Module |
427            +------------------+   Host   | Machine +------------------+
428   -------------------------------------- | ------------------------------
429             +---------------+            |          +---------------+
430             | PCI PF Device |            |          | PCI VF Device |
431             +---------------+            |          +---------------+
432
433FPGA PCIe device driver is always loaded first once an FPGA PCIe PF or VF device
434is detected. It:
435
436* Finishes enumeration on both FPGA PCIe PF and VF device using common
437  interfaces from DFL framework.
438* Supports SRIOV.
439
440The FME device driver plays a management role in this driver architecture, it
441provides ioctls to release Port from PF and assign Port to PF. After release
442a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV
443sysfs interface.
444
445To enable accessing an accelerator from applications running in a VM, the
446respective AFU's port needs to be assigned to a VF using the following steps:
447
448#. The PF owns all AFU ports by default. Any port that needs to be
449   reassigned to a VF must first be released through the
450   DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device.
451
452#. Once N ports are released from PF, then user can use command below
453   to enable SRIOV and VFs. Each VF owns only one Port with AFU.
454
455   ::
456
457      echo N > $PCI_DEVICE_PATH/sriov_numvfs
458
459#. Pass through the VFs to VMs
460
461#. The AFU under VF is accessible from applications in VM (using the
462   same driver inside the VF).
463
464Note that an FME can't be assigned to a VF, thus PR and other management
465functions are only available via the PF.
466
467Device enumeration
468==================
469This section introduces how applications enumerate the fpga device from
470the sysfs hierarchy under /sys/class/fpga_region.
471
472In the example below, two DFL based FPGA devices are installed in the host. Each
473fpga device has one FME and two ports (AFUs).
474
475FPGA regions are created under /sys/class/fpga_region/::
476
477	/sys/class/fpga_region/region0
478	/sys/class/fpga_region/region1
479	/sys/class/fpga_region/region2
480	...
481
482Application needs to search each regionX folder, if feature device is found,
483(e.g. "dfl-port.n" or "dfl-fme.m" is found), then it's the base
484fpga region which represents the FPGA device.
485
486Each base region has one FME and two ports (AFUs) as child devices::
487
488	/sys/class/fpga_region/region0/dfl-fme.0
489	/sys/class/fpga_region/region0/dfl-port.0
490	/sys/class/fpga_region/region0/dfl-port.1
491	...
492
493	/sys/class/fpga_region/region3/dfl-fme.1
494	/sys/class/fpga_region/region3/dfl-port.2
495	/sys/class/fpga_region/region3/dfl-port.3
496	...
497
498In general, the FME/AFU sysfs interfaces are named as follows::
499
500	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/
501	/sys/class/fpga_region/<regionX>/<dfl-port.m>/
502
503with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
504ports.
505
506The device nodes used for ioctl() or mmap() can be referenced through::
507
508	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev
509	/sys/class/fpga_region/<regionX>/<dfl-port.n>/dev
510
511
512Performance Counters
513====================
514Performance reporting is one private feature implemented in FME. It could
515supports several independent, system-wide, device counter sets in hardware to
516monitor and count for performance events, including "basic", "cache", "fabric",
517"vtd" and "vtd_sip" counters. Users could use standard perf tool to monitor
518FPGA cache hit/miss rate, transaction number, interface clock counter of AFU
519and other FPGA performance events.
520
521Different FPGA devices may have different counter sets, depending on hardware
522implementation. E.g., some discrete FPGA cards don't have any cache. User could
523use "perf list" to check which perf events are supported by target hardware.
524
525In order to allow user to use standard perf API to access these performance
526counters, driver creates a perf PMU, and related sysfs interfaces in
527/sys/bus/event_source/devices/dfl_fme* to describe available perf events and
528configuration options.
529
530The "format" directory describes the format of the config field of struct
531perf_event_attr. There are 3 bitfields for config: "evtype" defines which type
532the perf event belongs to; "event" is the identity of the event within its
533category; "portid" is introduced to decide counters set to monitor on FPGA
534overall data or a specific port.
535
536The "events" directory describes the configuration templates for all available
537events which can be used with perf tool directly. For example, fab_mmio_read
538has the configuration "event=0x06,evtype=0x02,portid=0xff", which shows this
539event belongs to fabric type (0x02), the local event id is 0x06 and it is for
540overall monitoring (portid=0xff).
541
542Example usage of perf::
543
544  $# perf list |grep dfl_fme
545
546  dfl_fme0/fab_mmio_read/                              [Kernel PMU event]
547  <...>
548  dfl_fme0/fab_port_mmio_read,portid=?/                [Kernel PMU event]
549  <...>
550
551  $# perf stat -a -e dfl_fme0/fab_mmio_read/ <command>
552  or
553  $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0xff/ <command>
554  or
555  $# perf stat -a -e dfl_fme0/config=0xff2006/ <command>
556
557Another example, fab_port_mmio_read monitors mmio read of a specific port. So
558its configuration template is "event=0x06,evtype=0x01,portid=?". The portid
559should be explicitly set.
560
561Its usage of perf::
562
563  $# perf stat -a -e dfl_fme0/fab_port_mmio_read,portid=0x0/ <command>
564  or
565  $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0x0/ <command>
566  or
567  $# perf stat -a -e dfl_fme0/config=0x2006/ <command>
568
569Please note for fabric counters, overall perf events (fab_*) and port perf
570events (fab_port_*) actually share one set of counters in hardware, so it can't
571monitor both at the same time. If this set of counters is configured to monitor
572overall data, then per port perf data is not supported. See below example::
573
574  $# perf stat -e dfl_fme0/fab_mmio_read/,dfl_fme0/fab_port_mmio_write,\
575                                                    portid=0/ sleep 1
576
577  Performance counter stats for 'system wide':
578
579                 3      dfl_fme0/fab_mmio_read/
580   <not supported>      dfl_fme0/fab_port_mmio_write,portid=0x0/
581
582       1.001750904 seconds time elapsed
583
584The driver also provides a "cpumask" sysfs attribute, which contains only one
585CPU id used to access these perf events. Counting on multiple CPU is not allowed
586since they are system-wide counters on FPGA device.
587
588The current driver does not support sampling. So "perf record" is unsupported.
589
590
591Interrupt support
592=================
593Some FME and AFU private features are able to generate interrupts. As mentioned
594above, users could call ioctl (DFL_FPGA_*_GET_IRQ_NUM) to know whether or how
595many interrupts are supported for this private feature. Drivers also implement
596an eventfd based interrupt handling mechanism for users to get notified when
597interrupt happens. Users could set eventfds to driver via
598ioctl (DFL_FPGA_*_SET_IRQ), and then poll/select on these eventfds waiting for
599notification.
600In Current DFL, 3 sub features (Port error, FME global error and AFU interrupt)
601support interrupts.
602
603
604Add new FIUs support
605====================
606It's possible that developers made some new function blocks (FIUs) under this
607DFL framework, then new platform device driver needs to be developed for the
608new feature dev (FIU) following the same way as existing feature dev drivers
609(e.g. FME and Port/AFU platform device driver). Besides that, it requires
610modification on DFL framework enumeration code too, for new FIU type detection
611and related platform devices creation.
612
613
614Add new private features support
615================================
616In some cases, we may need to add some new private features to existing FIUs
617(e.g. FME or Port). Developers don't need to touch enumeration code in DFL
618framework, as each private feature will be parsed automatically and related
619mmio resources can be found under FIU platform device created by DFL framework.
620Developer only needs to provide a sub feature driver with matched feature id.
621FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c)
622could be a reference.
623
624Please refer to below link to existing feature id table and guide for new feature
625ids application.
626https://github.com/OPAE/dfl-feature-id
627
628
629Location of DFLs on a PCI Device
630================================
631The original method for finding a DFL on a PCI device assumed the start of the
632first DFL to offset 0 of bar 0.  If the first node of the DFL is an FME,
633then further DFLs in the port(s) are specified in FME header registers.
634Alternatively, a PCIe vendor specific capability structure can be used to
635specify the location of all the DFLs on the device, providing flexibility
636for the type of starting node in the DFL.  Intel has reserved the
637VSEC ID of 0x43 for this purpose.  The vendor specific
638data begins with a 4 byte vendor specific register for the number of DFLs followed 4 byte
639Offset/BIR vendor specific registers for each DFL. Bits 2:0 of Offset/BIR register
640indicates the BAR, and bits 31:3 form the 8 byte aligned offset where bits 2:0 are
641zero.
642::
643
644        +----------------------------+
645        |31     Number of DFLS      0|
646        +----------------------------+
647        |31     Offset     3|2 BIR  0|
648        +----------------------------+
649                      . . .
650        +----------------------------+
651        |31     Offset     3|2 BIR  0|
652        +----------------------------+
653
654Being able to specify more than one DFL per BAR has been considered, but it
655was determined the use case did not provide value.  Specifying a single DFL
656per BAR simplifies the implementation and allows for extra error checking.
657
658
659Userspace driver support for DFL devices
660========================================
661The purpose of an FPGA is to be reprogrammed with newly developed hardware
662components. New hardware can instantiate a new private feature in the DFL, and
663then present a DFL device in the system. In some cases users may need a
664userspace driver for the DFL device:
665
666* Users may need to run some diagnostic test for their hardware.
667* Users may prototype the kernel driver in user space.
668* Some hardware is designed for specific purposes and does not fit into one of
669  the standard kernel subsystems.
670
671This requires direct access to MMIO space and interrupt handling from
672userspace. The uio_dfl module exposes the UIO device interfaces for this
673purpose.
674
675Currently the uio_dfl driver only supports the Ether Group sub feature, which
676has no irq in hardware. So the interrupt handling is not added in this driver.
677
678UIO_DFL should be selected to enable the uio_dfl module driver. To support a
679new DFL feature via UIO direct access, its feature id should be added to the
680driver's id_table.
681
682
683Open discussion
684===============
685FME driver exports one ioctl (DFL_FPGA_FME_PORT_PR) for partial reconfiguration
686to user now. In the future, if unified user interfaces for reconfiguration are
687added, FME driver should switch to them from ioctl interface.
688