1================================================= 2FPGA Device Feature List (DFL) Framework Overview 3================================================= 4 5Authors: 6 7- Enno Luebbers <enno.luebbers@intel.com> 8- Xiao Guangrong <guangrong.xiao@linux.intel.com> 9- Wu Hao <hao.wu@intel.com> 10 11The Device Feature List (DFL) FPGA framework (and drivers according to this 12this framework) hides the very details of low layer hardwares and provides 13unified interfaces to userspace. Applications could use these interfaces to 14configure, enumerate, open and access FPGA accelerators on platforms which 15implement the DFL in the device memory. Besides this, the DFL framework 16enables system level management functions such as FPGA reconfiguration. 17 18 19Device Feature List (DFL) Overview 20================================== 21Device Feature List (DFL) defines a linked list of feature headers within the 22device MMIO space to provide an extensible way of adding features. Software can 23walk through these predefined data structures to enumerate FPGA features: 24FPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features, 25as illustrated below:: 26 27 Header Header Header Header 28 +----------+ +-->+----------+ +-->+----------+ +-->+----------+ 29 | Type | | | Type | | | Type | | | Type | 30 | FIU | | | Private | | | Private | | | Private | 31 +----------+ | | Feature | | | Feature | | | Feature | 32 | Next_DFH |--+ +----------+ | +----------+ | +----------+ 33 +----------+ | Next_DFH |--+ | Next_DFH |--+ | Next_DFH |--> NULL 34 | ID | +----------+ +----------+ +----------+ 35 +----------+ | ID | | ID | | ID | 36 | Next_AFU |--+ +----------+ +----------+ +----------+ 37 +----------+ | | Feature | | Feature | | Feature | 38 | Header | | | Register | | Register | | Register | 39 | Register | | | Set | | Set | | Set | 40 | Set | | +----------+ +----------+ +----------+ 41 +----------+ | Header 42 +-->+----------+ 43 | Type | 44 | AFU | 45 +----------+ 46 | Next_DFH |--> NULL 47 +----------+ 48 | GUID | 49 +----------+ 50 | Header | 51 | Register | 52 | Set | 53 +----------+ 54 55FPGA Interface Unit (FIU) represents a standalone functional unit for the 56interface to FPGA, e.g. the FPGA Management Engine (FME) and Port (more 57descriptions on FME and Port in later sections). 58 59Accelerated Function Unit (AFU) represents a FPGA programmable region and 60always connects to a FIU (e.g. a Port) as its child as illustrated above. 61 62Private Features represent sub features of the FIU and AFU. They could be 63various function blocks with different IDs, but all private features which 64belong to the same FIU or AFU, must be linked to one list via the Next Device 65Feature Header (Next_DFH) pointer. 66 67Each FIU, AFU and Private Feature could implement its own functional registers. 68The functional register set for FIU and AFU, is named as Header Register Set, 69e.g. FME Header Register Set, and the one for Private Feature, is named as 70Feature Register Set, e.g. FME Partial Reconfiguration Feature Register Set. 71 72This Device Feature List provides a way of linking features together, it's 73convenient for software to locate each feature by walking through this list, 74and can be implemented in register regions of any FPGA device. 75 76 77FIU - FME (FPGA Management Engine) 78================================== 79The FPGA Management Engine performs reconfiguration and other infrastructure 80functions. Each FPGA device only has one FME. 81 82User-space applications can acquire exclusive access to the FME using open(), 83and release it using close(). 84 85The following functions are exposed through ioctls: 86 87- Get driver API version (DFL_FPGA_GET_API_VERSION) 88- Check for extensions (DFL_FPGA_CHECK_EXTENSION) 89- Program bitstream (DFL_FPGA_FME_PORT_PR) 90- Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN) 91- Release port from PF (DFL_FPGA_FME_PORT_RELEASE) 92 93More functions are exposed through sysfs 94(/sys/class/fpga_region/regionX/dfl-fme.n/): 95 96 Read bitstream ID (bitstream_id) 97 bitstream_id indicates version of the static FPGA region. 98 99 Read bitstream metadata (bitstream_metadata) 100 bitstream_metadata includes detailed information of static FPGA region, 101 e.g. synthesis date and seed. 102 103 Read number of ports (ports_num) 104 one FPGA device may have more than one port, this sysfs interface indicates 105 how many ports the FPGA device has. 106 107 Global error reporting management (errors/) 108 error reporting sysfs interfaces allow user to read errors detected by the 109 hardware, and clear the logged errors. 110 111 112FIU - PORT 113========== 114A port represents the interface between the static FPGA fabric and a partially 115reconfigurable region containing an AFU. It controls the communication from SW 116to the accelerator and exposes features such as reset and debug. Each FPGA 117device may have more than one port, but always one AFU per port. 118 119 120AFU 121=== 122An AFU is attached to a port FIU and exposes a fixed length MMIO region to be 123used for accelerator-specific control registers. 124 125User-space applications can acquire exclusive access to an AFU attached to a 126port by using open() on the port device node and release it using close(). 127 128The following functions are exposed through ioctls: 129 130- Get driver API version (DFL_FPGA_GET_API_VERSION) 131- Check for extensions (DFL_FPGA_CHECK_EXTENSION) 132- Get port info (DFL_FPGA_PORT_GET_INFO) 133- Get MMIO region info (DFL_FPGA_PORT_GET_REGION_INFO) 134- Map DMA buffer (DFL_FPGA_PORT_DMA_MAP) 135- Unmap DMA buffer (DFL_FPGA_PORT_DMA_UNMAP) 136- Reset AFU (DFL_FPGA_PORT_RESET) 137 138DFL_FPGA_PORT_RESET: 139 reset the FPGA Port and its AFU. Userspace can do Port 140 reset at any time, e.g. during DMA or Partial Reconfiguration. But it should 141 never cause any system level issue, only functional failure (e.g. DMA or PR 142 operation failure) and be recoverable from the failure. 143 144User-space applications can also mmap() accelerator MMIO regions. 145 146More functions are exposed through sysfs: 147(/sys/class/fpga_region/<regionX>/<dfl-port.m>/): 148 149 Read Accelerator GUID (afu_id) 150 afu_id indicates which PR bitstream is programmed to this AFU. 151 152 Error reporting (errors/) 153 error reporting sysfs interfaces allow user to read port/afu errors 154 detected by the hardware, and clear the logged errors. 155 156 157DFL Framework Overview 158====================== 159 160:: 161 162 +----------+ +--------+ +--------+ +--------+ 163 | FME | | AFU | | AFU | | AFU | 164 | Module | | Module | | Module | | Module | 165 +----------+ +--------+ +--------+ +--------+ 166 +-----------------------+ 167 | FPGA Container Device | Device Feature List 168 | (FPGA Base Region) | Framework 169 +-----------------------+ 170 ------------------------------------------------------------------ 171 +----------------------------+ 172 | FPGA DFL Device Module | 173 | (e.g. PCIE/Platform Device)| 174 +----------------------------+ 175 +------------------------+ 176 | FPGA Hardware Device | 177 +------------------------+ 178 179DFL framework in kernel provides common interfaces to create container device 180(FPGA base region), discover feature devices and their private features from the 181given Device Feature Lists and create platform devices for feature devices 182(e.g. FME, Port and AFU) with related resources under the container device. It 183also abstracts operations for the private features and exposes common ops to 184feature device drivers. 185 186The FPGA DFL Device could be different hardwares, e.g. PCIe device, platform 187device and etc. Its driver module is always loaded first once the device is 188created by the system. This driver plays an infrastructural role in the 189driver architecture. It locates the DFLs in the device memory, handles them 190and related resources to common interfaces from DFL framework for enumeration. 191(Please refer to drivers/fpga/dfl.c for detailed enumeration APIs). 192 193The FPGA Management Engine (FME) driver is a platform driver which is loaded 194automatically after FME platform device creation from the DFL device module. It 195provides the key features for FPGA management, including: 196 197 a) Expose static FPGA region information, e.g. version and metadata. 198 Users can read related information via sysfs interfaces exposed 199 by FME driver. 200 201 b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA 202 bridges and FPGA regions during PR sub feature initialization. Once 203 it receives a DFL_FPGA_FME_PORT_PR ioctl from user, it invokes the 204 common interface function from FPGA Region to complete the partial 205 reconfiguration of the PR bitstream to the given port. 206 207Similar to the FME driver, the FPGA Accelerated Function Unit (AFU) driver is 208probed once the AFU platform device is created. The main function of this module 209is to provide an interface for userspace applications to access the individual 210accelerators, including basic reset control on port, AFU MMIO region export, dma 211buffer mapping service functions. 212 213After feature platform devices creation, matched platform drivers will be loaded 214automatically to handle different functionalities. Please refer to next sections 215for detailed information on functional units which have been already implemented 216under this DFL framework. 217 218 219Partial Reconfiguration 220======================= 221As mentioned above, accelerators can be reconfigured through partial 222reconfiguration of a PR bitstream file. The PR bitstream file must have been 223generated for the exact static FPGA region and targeted reconfigurable region 224(port) of the FPGA, otherwise, the reconfiguration operation will fail and 225possibly cause system instability. This compatibility can be checked by 226comparing the compatibility ID noted in the header of PR bitstream file against 227the compat_id exposed by the target FPGA region. This check is usually done by 228userspace before calling the reconfiguration IOCTL. 229 230 231FPGA virtualization - PCIe SRIOV 232================================ 233This section describes the virtualization support on DFL based FPGA device to 234enable accessing an accelerator from applications running in a virtual machine 235(VM). This section only describes the PCIe based FPGA device with SRIOV support. 236 237Features supported by the particular FPGA device are exposed through Device 238Feature Lists, as illustrated below: 239 240:: 241 242 +-------------------------------+ +-------------+ 243 | PF | | VF | 244 +-------------------------------+ +-------------+ 245 ^ ^ ^ ^ 246 | | | | 247 +-----|------------|---------|--------------|-------+ 248 | | | | | | 249 | +-----+ +-------+ +-------+ +-------+ | 250 | | FME | | Port0 | | Port1 | | Port2 | | 251 | +-----+ +-------+ +-------+ +-------+ | 252 | ^ ^ ^ | 253 | | | | | 254 | +-------+ +------+ +-------+ | 255 | | AFU | | AFU | | AFU | | 256 | +-------+ +------+ +-------+ | 257 | | 258 | DFL based FPGA PCIe Device | 259 +---------------------------------------------------+ 260 261FME is always accessed through the physical function (PF). 262 263Ports (and related AFUs) are accessed via PF by default, but could be exposed 264through virtual function (VF) devices via PCIe SRIOV. Each VF only contains 2651 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators) 266created via PCIe SRIOV interface, to virtual machines. 267 268The driver organization in virtualization case is illustrated below: 269:: 270 271 +-------++------++------+ | 272 | FME || FME || FME | | 273 | FPGA || FPGA || FPGA | | 274 |Manager||Bridge||Region| | 275 +-------++------++------+ | 276 +-----------------------+ +--------+ | +--------+ 277 | FME | | AFU | | | AFU | 278 | Module | | Module | | | Module | 279 +-----------------------+ +--------+ | +--------+ 280 +-----------------------+ | +-----------------------+ 281 | FPGA Container Device | | | FPGA Container Device | 282 | (FPGA Base Region) | | | (FPGA Base Region) | 283 +-----------------------+ | +-----------------------+ 284 +------------------+ | +------------------+ 285 | FPGA PCIE Module | | Virtual | FPGA PCIE Module | 286 +------------------+ Host | Machine +------------------+ 287 -------------------------------------- | ------------------------------ 288 +---------------+ | +---------------+ 289 | PCI PF Device | | | PCI VF Device | 290 +---------------+ | +---------------+ 291 292FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device 293is detected. It: 294 295* Finishes enumeration on both FPGA PCIe PF and VF device using common 296 interfaces from DFL framework. 297* Supports SRIOV. 298 299The FME device driver plays a management role in this driver architecture, it 300provides ioctls to release Port from PF and assign Port to PF. After release 301a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV 302sysfs interface. 303 304To enable accessing an accelerator from applications running in a VM, the 305respective AFU's port needs to be assigned to a VF using the following steps: 306 307#. The PF owns all AFU ports by default. Any port that needs to be 308 reassigned to a VF must first be released through the 309 DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device. 310 311#. Once N ports are released from PF, then user can use command below 312 to enable SRIOV and VFs. Each VF owns only one Port with AFU. 313 314 :: 315 316 echo N > $PCI_DEVICE_PATH/sriov_numvfs 317 318#. Pass through the VFs to VMs 319 320#. The AFU under VF is accessible from applications in VM (using the 321 same driver inside the VF). 322 323Note that an FME can't be assigned to a VF, thus PR and other management 324functions are only available via the PF. 325 326Device enumeration 327================== 328This section introduces how applications enumerate the fpga device from 329the sysfs hierarchy under /sys/class/fpga_region. 330 331In the example below, two DFL based FPGA devices are installed in the host. Each 332fpga device has one FME and two ports (AFUs). 333 334FPGA regions are created under /sys/class/fpga_region/:: 335 336 /sys/class/fpga_region/region0 337 /sys/class/fpga_region/region1 338 /sys/class/fpga_region/region2 339 ... 340 341Application needs to search each regionX folder, if feature device is found, 342(e.g. "dfl-port.n" or "dfl-fme.m" is found), then it's the base 343fpga region which represents the FPGA device. 344 345Each base region has one FME and two ports (AFUs) as child devices:: 346 347 /sys/class/fpga_region/region0/dfl-fme.0 348 /sys/class/fpga_region/region0/dfl-port.0 349 /sys/class/fpga_region/region0/dfl-port.1 350 ... 351 352 /sys/class/fpga_region/region3/dfl-fme.1 353 /sys/class/fpga_region/region3/dfl-port.2 354 /sys/class/fpga_region/region3/dfl-port.3 355 ... 356 357In general, the FME/AFU sysfs interfaces are named as follows:: 358 359 /sys/class/fpga_region/<regionX>/<dfl-fme.n>/ 360 /sys/class/fpga_region/<regionX>/<dfl-port.m>/ 361 362with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all 363ports. 364 365The device nodes used for ioctl() or mmap() can be referenced through:: 366 367 /sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev 368 /sys/class/fpga_region/<regionX>/<dfl-port.n>/dev 369 370 371Add new FIUs support 372==================== 373It's possible that developers made some new function blocks (FIUs) under this 374DFL framework, then new platform device driver needs to be developed for the 375new feature dev (FIU) following the same way as existing feature dev drivers 376(e.g. FME and Port/AFU platform device driver). Besides that, it requires 377modification on DFL framework enumeration code too, for new FIU type detection 378and related platform devices creation. 379 380 381Add new private features support 382================================ 383In some cases, we may need to add some new private features to existing FIUs 384(e.g. FME or Port). Developers don't need to touch enumeration code in DFL 385framework, as each private feature will be parsed automatically and related 386mmio resources can be found under FIU platform device created by DFL framework. 387Developer only needs to provide a sub feature driver with matched feature id. 388FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c) 389could be a reference. 390 391 392Open discussion 393=============== 394FME driver exports one ioctl (DFL_FPGA_FME_PORT_PR) for partial reconfiguration 395to user now. In the future, if unified user interfaces for reconfiguration are 396added, FME driver should switch to them from ioctl interface. 397