14d2e26a3SMauro Carvalho Chehab==================================== 24d2e26a3SMauro Carvalho ChehabCoherent Accelerator Interface (CXL) 34d2e26a3SMauro Carvalho Chehab==================================== 44d2e26a3SMauro Carvalho Chehab 54d2e26a3SMauro Carvalho ChehabIntroduction 64d2e26a3SMauro Carvalho Chehab============ 74d2e26a3SMauro Carvalho Chehab 84d2e26a3SMauro Carvalho Chehab The coherent accelerator interface is designed to allow the 94d2e26a3SMauro Carvalho Chehab coherent connection of accelerators (FPGAs and other devices) to a 104d2e26a3SMauro Carvalho Chehab POWER system. These devices need to adhere to the Coherent 114d2e26a3SMauro Carvalho Chehab Accelerator Interface Architecture (CAIA). 124d2e26a3SMauro Carvalho Chehab 134d2e26a3SMauro Carvalho Chehab IBM refers to this as the Coherent Accelerator Processor Interface 144d2e26a3SMauro Carvalho Chehab or CAPI. In the kernel it's referred to by the name CXL to avoid 154d2e26a3SMauro Carvalho Chehab confusion with the ISDN CAPI subsystem. 164d2e26a3SMauro Carvalho Chehab 174d2e26a3SMauro Carvalho Chehab Coherent in this context means that the accelerator and CPUs can 184d2e26a3SMauro Carvalho Chehab both access system memory directly and with the same effective 194d2e26a3SMauro Carvalho Chehab addresses. 204d2e26a3SMauro Carvalho Chehab 214d2e26a3SMauro Carvalho Chehab 224d2e26a3SMauro Carvalho ChehabHardware overview 234d2e26a3SMauro Carvalho Chehab================= 244d2e26a3SMauro Carvalho Chehab 254d2e26a3SMauro Carvalho Chehab :: 264d2e26a3SMauro Carvalho Chehab 274d2e26a3SMauro Carvalho Chehab POWER8/9 FPGA 284d2e26a3SMauro Carvalho Chehab +----------+ +---------+ 294d2e26a3SMauro Carvalho Chehab | | | | 304d2e26a3SMauro Carvalho Chehab | CPU | | AFU | 314d2e26a3SMauro Carvalho Chehab | | | | 324d2e26a3SMauro Carvalho Chehab | | | | 334d2e26a3SMauro Carvalho Chehab | | | | 344d2e26a3SMauro Carvalho Chehab +----------+ +---------+ 354d2e26a3SMauro Carvalho Chehab | PHB | | | 364d2e26a3SMauro Carvalho Chehab | +------+ | PSL | 374d2e26a3SMauro Carvalho Chehab | | CAPP |<------>| | 384d2e26a3SMauro Carvalho Chehab +---+------+ PCIE +---------+ 394d2e26a3SMauro Carvalho Chehab 404d2e26a3SMauro Carvalho Chehab The POWER8/9 chip has a Coherently Attached Processor Proxy (CAPP) 414d2e26a3SMauro Carvalho Chehab unit which is part of the PCIe Host Bridge (PHB). This is managed 424d2e26a3SMauro Carvalho Chehab by Linux by calls into OPAL. Linux doesn't directly program the 434d2e26a3SMauro Carvalho Chehab CAPP. 444d2e26a3SMauro Carvalho Chehab 454d2e26a3SMauro Carvalho Chehab The FPGA (or coherently attached device) consists of two parts. 464d2e26a3SMauro Carvalho Chehab The POWER Service Layer (PSL) and the Accelerator Function Unit 474d2e26a3SMauro Carvalho Chehab (AFU). The AFU is used to implement specific functionality behind 484d2e26a3SMauro Carvalho Chehab the PSL. The PSL, among other things, provides memory address 494d2e26a3SMauro Carvalho Chehab translation services to allow each AFU direct access to userspace 504d2e26a3SMauro Carvalho Chehab memory. 514d2e26a3SMauro Carvalho Chehab 524d2e26a3SMauro Carvalho Chehab The AFU is the core part of the accelerator (eg. the compression, 534d2e26a3SMauro Carvalho Chehab crypto etc function). The kernel has no knowledge of the function 544d2e26a3SMauro Carvalho Chehab of the AFU. Only userspace interacts directly with the AFU. 554d2e26a3SMauro Carvalho Chehab 564d2e26a3SMauro Carvalho Chehab The PSL provides the translation and interrupt services that the 574d2e26a3SMauro Carvalho Chehab AFU needs. This is what the kernel interacts with. For example, if 584d2e26a3SMauro Carvalho Chehab the AFU needs to read a particular effective address, it sends 594d2e26a3SMauro Carvalho Chehab that address to the PSL, the PSL then translates it, fetches the 604d2e26a3SMauro Carvalho Chehab data from memory and returns it to the AFU. If the PSL has a 614d2e26a3SMauro Carvalho Chehab translation miss, it interrupts the kernel and the kernel services 624d2e26a3SMauro Carvalho Chehab the fault. The context to which this fault is serviced is based on 634d2e26a3SMauro Carvalho Chehab who owns that acceleration function. 644d2e26a3SMauro Carvalho Chehab 654d2e26a3SMauro Carvalho Chehab - POWER8 and PSL Version 8 are compliant to the CAIA Version 1.0. 664d2e26a3SMauro Carvalho Chehab - POWER9 and PSL Version 9 are compliant to the CAIA Version 2.0. 674d2e26a3SMauro Carvalho Chehab 684d2e26a3SMauro Carvalho Chehab This PSL Version 9 provides new features such as: 694d2e26a3SMauro Carvalho Chehab 704d2e26a3SMauro Carvalho Chehab * Interaction with the nest MMU on the P9 chip. 714d2e26a3SMauro Carvalho Chehab * Native DMA support. 724d2e26a3SMauro Carvalho Chehab * Supports sending ASB_Notify messages for host thread wakeup. 734d2e26a3SMauro Carvalho Chehab * Supports Atomic operations. 744d2e26a3SMauro Carvalho Chehab * etc. 754d2e26a3SMauro Carvalho Chehab 764d2e26a3SMauro Carvalho Chehab Cards with a PSL9 won't work on a POWER8 system and cards with a 774d2e26a3SMauro Carvalho Chehab PSL8 won't work on a POWER9 system. 784d2e26a3SMauro Carvalho Chehab 794d2e26a3SMauro Carvalho ChehabAFU Modes 804d2e26a3SMauro Carvalho Chehab========= 814d2e26a3SMauro Carvalho Chehab 824d2e26a3SMauro Carvalho Chehab There are two programming modes supported by the AFU. Dedicated 834d2e26a3SMauro Carvalho Chehab and AFU directed. AFU may support one or both modes. 844d2e26a3SMauro Carvalho Chehab 854d2e26a3SMauro Carvalho Chehab When using dedicated mode only one MMU context is supported. In 864d2e26a3SMauro Carvalho Chehab this mode, only one userspace process can use the accelerator at 874d2e26a3SMauro Carvalho Chehab time. 884d2e26a3SMauro Carvalho Chehab 894d2e26a3SMauro Carvalho Chehab When using AFU directed mode, up to 16K simultaneous contexts can 904d2e26a3SMauro Carvalho Chehab be supported. This means up to 16K simultaneous userspace 914d2e26a3SMauro Carvalho Chehab applications may use the accelerator (although specific AFUs may 924d2e26a3SMauro Carvalho Chehab support fewer). In this mode, the AFU sends a 16 bit context ID 934d2e26a3SMauro Carvalho Chehab with each of its requests. This tells the PSL which context is 944d2e26a3SMauro Carvalho Chehab associated with each operation. If the PSL can't translate an 954d2e26a3SMauro Carvalho Chehab operation, the ID can also be accessed by the kernel so it can 964d2e26a3SMauro Carvalho Chehab determine the userspace context associated with an operation. 974d2e26a3SMauro Carvalho Chehab 984d2e26a3SMauro Carvalho Chehab 994d2e26a3SMauro Carvalho ChehabMMIO space 1004d2e26a3SMauro Carvalho Chehab========== 1014d2e26a3SMauro Carvalho Chehab 1024d2e26a3SMauro Carvalho Chehab A portion of the accelerator MMIO space can be directly mapped 1034d2e26a3SMauro Carvalho Chehab from the AFU to userspace. Either the whole space can be mapped or 1044d2e26a3SMauro Carvalho Chehab just a per context portion. The hardware is self describing, hence 1054d2e26a3SMauro Carvalho Chehab the kernel can determine the offset and size of the per context 1064d2e26a3SMauro Carvalho Chehab portion. 1074d2e26a3SMauro Carvalho Chehab 1084d2e26a3SMauro Carvalho Chehab 1094d2e26a3SMauro Carvalho ChehabInterrupts 1104d2e26a3SMauro Carvalho Chehab========== 1114d2e26a3SMauro Carvalho Chehab 1124d2e26a3SMauro Carvalho Chehab AFUs may generate interrupts that are destined for userspace. These 1134d2e26a3SMauro Carvalho Chehab are received by the kernel as hardware interrupts and passed onto 1144d2e26a3SMauro Carvalho Chehab userspace by a read syscall documented below. 1154d2e26a3SMauro Carvalho Chehab 1164d2e26a3SMauro Carvalho Chehab Data storage faults and error interrupts are handled by the kernel 1174d2e26a3SMauro Carvalho Chehab driver. 1184d2e26a3SMauro Carvalho Chehab 1194d2e26a3SMauro Carvalho Chehab 1204d2e26a3SMauro Carvalho ChehabWork Element Descriptor (WED) 1214d2e26a3SMauro Carvalho Chehab============================= 1224d2e26a3SMauro Carvalho Chehab 1234d2e26a3SMauro Carvalho Chehab The WED is a 64-bit parameter passed to the AFU when a context is 1244d2e26a3SMauro Carvalho Chehab started. Its format is up to the AFU hence the kernel has no 1254d2e26a3SMauro Carvalho Chehab knowledge of what it represents. Typically it will be the 1264d2e26a3SMauro Carvalho Chehab effective address of a work queue or status block where the AFU 1274d2e26a3SMauro Carvalho Chehab and userspace can share control and status information. 1284d2e26a3SMauro Carvalho Chehab 1294d2e26a3SMauro Carvalho Chehab 1304d2e26a3SMauro Carvalho Chehab 1314d2e26a3SMauro Carvalho Chehab 1324d2e26a3SMauro Carvalho ChehabUser API 1334d2e26a3SMauro Carvalho Chehab======== 1344d2e26a3SMauro Carvalho Chehab 1354d2e26a3SMauro Carvalho Chehab1. AFU character devices 1368f97986cSMauro Carvalho Chehab^^^^^^^^^^^^^^^^^^^^^^^^ 1374d2e26a3SMauro Carvalho Chehab 1384d2e26a3SMauro Carvalho Chehab For AFUs operating in AFU directed mode, two character device 1394d2e26a3SMauro Carvalho Chehab files will be created. /dev/cxl/afu0.0m will correspond to a 1404d2e26a3SMauro Carvalho Chehab master context and /dev/cxl/afu0.0s will correspond to a slave 1414d2e26a3SMauro Carvalho Chehab context. Master contexts have access to the full MMIO space an 1424d2e26a3SMauro Carvalho Chehab AFU provides. Slave contexts have access to only the per process 1434d2e26a3SMauro Carvalho Chehab MMIO space an AFU provides. 1444d2e26a3SMauro Carvalho Chehab 1454d2e26a3SMauro Carvalho Chehab For AFUs operating in dedicated process mode, the driver will 1464d2e26a3SMauro Carvalho Chehab only create a single character device per AFU called 1474d2e26a3SMauro Carvalho Chehab /dev/cxl/afu0.0d. This will have access to the entire MMIO space 1484d2e26a3SMauro Carvalho Chehab that the AFU provides (like master contexts in AFU directed). 1494d2e26a3SMauro Carvalho Chehab 1504d2e26a3SMauro Carvalho Chehab The types described below are defined in include/uapi/misc/cxl.h 1514d2e26a3SMauro Carvalho Chehab 1524d2e26a3SMauro Carvalho Chehab The following file operations are supported on both slave and 1534d2e26a3SMauro Carvalho Chehab master devices. 1544d2e26a3SMauro Carvalho Chehab 1554d2e26a3SMauro Carvalho Chehab A userspace library libcxl is available here: 1564d2e26a3SMauro Carvalho Chehab 1574d2e26a3SMauro Carvalho Chehab https://github.com/ibm-capi/libcxl 1584d2e26a3SMauro Carvalho Chehab 1594d2e26a3SMauro Carvalho Chehab This provides a C interface to this kernel API. 1604d2e26a3SMauro Carvalho Chehab 1614d2e26a3SMauro Carvalho Chehabopen 1624d2e26a3SMauro Carvalho Chehab---- 1634d2e26a3SMauro Carvalho Chehab 1644d2e26a3SMauro Carvalho Chehab Opens the device and allocates a file descriptor to be used with 1654d2e26a3SMauro Carvalho Chehab the rest of the API. 1664d2e26a3SMauro Carvalho Chehab 1674d2e26a3SMauro Carvalho Chehab A dedicated mode AFU only has one context and only allows the 1684d2e26a3SMauro Carvalho Chehab device to be opened once. 1694d2e26a3SMauro Carvalho Chehab 1704d2e26a3SMauro Carvalho Chehab An AFU directed mode AFU can have many contexts, the device can be 1714d2e26a3SMauro Carvalho Chehab opened once for each context that is available. 1724d2e26a3SMauro Carvalho Chehab 1734d2e26a3SMauro Carvalho Chehab When all available contexts are allocated the open call will fail 1744d2e26a3SMauro Carvalho Chehab and return -ENOSPC. 1754d2e26a3SMauro Carvalho Chehab 1764d2e26a3SMauro Carvalho Chehab Note: 1774d2e26a3SMauro Carvalho Chehab IRQs need to be allocated for each context, which may limit 1784d2e26a3SMauro Carvalho Chehab the number of contexts that can be created, and therefore 1794d2e26a3SMauro Carvalho Chehab how many times the device can be opened. The POWER8 CAPP 1804d2e26a3SMauro Carvalho Chehab supports 2040 IRQs and 3 are used by the kernel, so 2037 are 1814d2e26a3SMauro Carvalho Chehab left. If 1 IRQ is needed per context, then only 2037 1824d2e26a3SMauro Carvalho Chehab contexts can be allocated. If 4 IRQs are needed per context, 1834d2e26a3SMauro Carvalho Chehab then only 2037/4 = 509 contexts can be allocated. 1844d2e26a3SMauro Carvalho Chehab 1854d2e26a3SMauro Carvalho Chehab 1864d2e26a3SMauro Carvalho Chehabioctl 1874d2e26a3SMauro Carvalho Chehab----- 1884d2e26a3SMauro Carvalho Chehab 1894d2e26a3SMauro Carvalho Chehab CXL_IOCTL_START_WORK: 1904d2e26a3SMauro Carvalho Chehab Starts the AFU context and associates it with the current 1914d2e26a3SMauro Carvalho Chehab process. Once this ioctl is successfully executed, all memory 1924d2e26a3SMauro Carvalho Chehab mapped into this process is accessible to this AFU context 1934d2e26a3SMauro Carvalho Chehab using the same effective addresses. No additional calls are 1944d2e26a3SMauro Carvalho Chehab required to map/unmap memory. The AFU memory context will be 1954d2e26a3SMauro Carvalho Chehab updated as userspace allocates and frees memory. This ioctl 1964d2e26a3SMauro Carvalho Chehab returns once the AFU context is started. 1974d2e26a3SMauro Carvalho Chehab 1984d2e26a3SMauro Carvalho Chehab Takes a pointer to a struct cxl_ioctl_start_work 1994d2e26a3SMauro Carvalho Chehab 2004d2e26a3SMauro Carvalho Chehab :: 2014d2e26a3SMauro Carvalho Chehab 2024d2e26a3SMauro Carvalho Chehab struct cxl_ioctl_start_work { 2034d2e26a3SMauro Carvalho Chehab __u64 flags; 2044d2e26a3SMauro Carvalho Chehab __u64 work_element_descriptor; 2054d2e26a3SMauro Carvalho Chehab __u64 amr; 2064d2e26a3SMauro Carvalho Chehab __s16 num_interrupts; 2074d2e26a3SMauro Carvalho Chehab __s16 reserved1; 2084d2e26a3SMauro Carvalho Chehab __s32 reserved2; 2094d2e26a3SMauro Carvalho Chehab __u64 reserved3; 2104d2e26a3SMauro Carvalho Chehab __u64 reserved4; 2114d2e26a3SMauro Carvalho Chehab __u64 reserved5; 2124d2e26a3SMauro Carvalho Chehab __u64 reserved6; 2134d2e26a3SMauro Carvalho Chehab }; 2144d2e26a3SMauro Carvalho Chehab 2154d2e26a3SMauro Carvalho Chehab flags: 2164d2e26a3SMauro Carvalho Chehab Indicates which optional fields in the structure are 2174d2e26a3SMauro Carvalho Chehab valid. 2184d2e26a3SMauro Carvalho Chehab 2194d2e26a3SMauro Carvalho Chehab work_element_descriptor: 2204d2e26a3SMauro Carvalho Chehab The Work Element Descriptor (WED) is a 64-bit argument 2214d2e26a3SMauro Carvalho Chehab defined by the AFU. Typically this is an effective 2224d2e26a3SMauro Carvalho Chehab address pointing to an AFU specific structure 2234d2e26a3SMauro Carvalho Chehab describing what work to perform. 2244d2e26a3SMauro Carvalho Chehab 2254d2e26a3SMauro Carvalho Chehab amr: 2264d2e26a3SMauro Carvalho Chehab Authority Mask Register (AMR), same as the powerpc 2274d2e26a3SMauro Carvalho Chehab AMR. This field is only used by the kernel when the 2284d2e26a3SMauro Carvalho Chehab corresponding CXL_START_WORK_AMR value is specified in 2294d2e26a3SMauro Carvalho Chehab flags. If not specified the kernel will use a default 2304d2e26a3SMauro Carvalho Chehab value of 0. 2314d2e26a3SMauro Carvalho Chehab 2324d2e26a3SMauro Carvalho Chehab num_interrupts: 2334d2e26a3SMauro Carvalho Chehab Number of userspace interrupts to request. This field 2344d2e26a3SMauro Carvalho Chehab is only used by the kernel when the corresponding 2354d2e26a3SMauro Carvalho Chehab CXL_START_WORK_NUM_IRQS value is specified in flags. 2364d2e26a3SMauro Carvalho Chehab If not specified the minimum number required by the 2374d2e26a3SMauro Carvalho Chehab AFU will be allocated. The min and max number can be 2384d2e26a3SMauro Carvalho Chehab obtained from sysfs. 2394d2e26a3SMauro Carvalho Chehab 2404d2e26a3SMauro Carvalho Chehab reserved fields: 2414d2e26a3SMauro Carvalho Chehab For ABI padding and future extensions 2424d2e26a3SMauro Carvalho Chehab 2434d2e26a3SMauro Carvalho Chehab CXL_IOCTL_GET_PROCESS_ELEMENT: 2444d2e26a3SMauro Carvalho Chehab Get the current context id, also known as the process element. 2454d2e26a3SMauro Carvalho Chehab The value is returned from the kernel as a __u32. 2464d2e26a3SMauro Carvalho Chehab 2474d2e26a3SMauro Carvalho Chehab 2484d2e26a3SMauro Carvalho Chehabmmap 2494d2e26a3SMauro Carvalho Chehab---- 2504d2e26a3SMauro Carvalho Chehab 2514d2e26a3SMauro Carvalho Chehab An AFU may have an MMIO space to facilitate communication with the 2524d2e26a3SMauro Carvalho Chehab AFU. If it does, the MMIO space can be accessed via mmap. The size 2534d2e26a3SMauro Carvalho Chehab and contents of this area are specific to the particular AFU. The 2544d2e26a3SMauro Carvalho Chehab size can be discovered via sysfs. 2554d2e26a3SMauro Carvalho Chehab 2564d2e26a3SMauro Carvalho Chehab In AFU directed mode, master contexts are allowed to map all of 2574d2e26a3SMauro Carvalho Chehab the MMIO space and slave contexts are allowed to only map the per 2584d2e26a3SMauro Carvalho Chehab process MMIO space associated with the context. In dedicated 2594d2e26a3SMauro Carvalho Chehab process mode the entire MMIO space can always be mapped. 2604d2e26a3SMauro Carvalho Chehab 2614d2e26a3SMauro Carvalho Chehab This mmap call must be done after the START_WORK ioctl. 2624d2e26a3SMauro Carvalho Chehab 2634d2e26a3SMauro Carvalho Chehab Care should be taken when accessing MMIO space. Only 32 and 64-bit 2644d2e26a3SMauro Carvalho Chehab accesses are supported by POWER8. Also, the AFU will be designed 2654d2e26a3SMauro Carvalho Chehab with a specific endianness, so all MMIO accesses should consider 2664d2e26a3SMauro Carvalho Chehab endianness (recommend endian(3) variants like: le64toh(), 2674d2e26a3SMauro Carvalho Chehab be64toh() etc). These endian issues equally apply to shared memory 2684d2e26a3SMauro Carvalho Chehab queues the WED may describe. 2694d2e26a3SMauro Carvalho Chehab 2704d2e26a3SMauro Carvalho Chehab 2714d2e26a3SMauro Carvalho Chehabread 2724d2e26a3SMauro Carvalho Chehab---- 2734d2e26a3SMauro Carvalho Chehab 2744d2e26a3SMauro Carvalho Chehab Reads events from the AFU. Blocks if no events are pending 2754d2e26a3SMauro Carvalho Chehab (unless O_NONBLOCK is supplied). Returns -EIO in the case of an 2764d2e26a3SMauro Carvalho Chehab unrecoverable error or if the card is removed. 2774d2e26a3SMauro Carvalho Chehab 2784d2e26a3SMauro Carvalho Chehab read() will always return an integral number of events. 2794d2e26a3SMauro Carvalho Chehab 2804d2e26a3SMauro Carvalho Chehab The buffer passed to read() must be at least 4K bytes. 2814d2e26a3SMauro Carvalho Chehab 2824d2e26a3SMauro Carvalho Chehab The result of the read will be a buffer of one or more events, 2834d2e26a3SMauro Carvalho Chehab each event is of type struct cxl_event, of varying size:: 2844d2e26a3SMauro Carvalho Chehab 2854d2e26a3SMauro Carvalho Chehab struct cxl_event { 2864d2e26a3SMauro Carvalho Chehab struct cxl_event_header header; 2874d2e26a3SMauro Carvalho Chehab union { 2884d2e26a3SMauro Carvalho Chehab struct cxl_event_afu_interrupt irq; 2894d2e26a3SMauro Carvalho Chehab struct cxl_event_data_storage fault; 2904d2e26a3SMauro Carvalho Chehab struct cxl_event_afu_error afu_error; 2914d2e26a3SMauro Carvalho Chehab }; 2924d2e26a3SMauro Carvalho Chehab }; 2934d2e26a3SMauro Carvalho Chehab 2944d2e26a3SMauro Carvalho Chehab The struct cxl_event_header is defined as 2954d2e26a3SMauro Carvalho Chehab 2964d2e26a3SMauro Carvalho Chehab :: 2974d2e26a3SMauro Carvalho Chehab 2984d2e26a3SMauro Carvalho Chehab struct cxl_event_header { 2994d2e26a3SMauro Carvalho Chehab __u16 type; 3004d2e26a3SMauro Carvalho Chehab __u16 size; 3014d2e26a3SMauro Carvalho Chehab __u16 process_element; 3024d2e26a3SMauro Carvalho Chehab __u16 reserved1; 3034d2e26a3SMauro Carvalho Chehab }; 3044d2e26a3SMauro Carvalho Chehab 3054d2e26a3SMauro Carvalho Chehab type: 3064d2e26a3SMauro Carvalho Chehab This defines the type of event. The type determines how 3074d2e26a3SMauro Carvalho Chehab the rest of the event is structured. These types are 3084d2e26a3SMauro Carvalho Chehab described below and defined by enum cxl_event_type. 3094d2e26a3SMauro Carvalho Chehab 3104d2e26a3SMauro Carvalho Chehab size: 3114d2e26a3SMauro Carvalho Chehab This is the size of the event in bytes including the 3124d2e26a3SMauro Carvalho Chehab struct cxl_event_header. The start of the next event can 3134d2e26a3SMauro Carvalho Chehab be found at this offset from the start of the current 3144d2e26a3SMauro Carvalho Chehab event. 3154d2e26a3SMauro Carvalho Chehab 3164d2e26a3SMauro Carvalho Chehab process_element: 3174d2e26a3SMauro Carvalho Chehab Context ID of the event. 3184d2e26a3SMauro Carvalho Chehab 3194d2e26a3SMauro Carvalho Chehab reserved field: 3204d2e26a3SMauro Carvalho Chehab For future extensions and padding. 3214d2e26a3SMauro Carvalho Chehab 3224d2e26a3SMauro Carvalho Chehab If the event type is CXL_EVENT_AFU_INTERRUPT then the event 3234d2e26a3SMauro Carvalho Chehab structure is defined as 3244d2e26a3SMauro Carvalho Chehab 3254d2e26a3SMauro Carvalho Chehab :: 3264d2e26a3SMauro Carvalho Chehab 3274d2e26a3SMauro Carvalho Chehab struct cxl_event_afu_interrupt { 3284d2e26a3SMauro Carvalho Chehab __u16 flags; 3294d2e26a3SMauro Carvalho Chehab __u16 irq; /* Raised AFU interrupt number */ 3304d2e26a3SMauro Carvalho Chehab __u32 reserved1; 3314d2e26a3SMauro Carvalho Chehab }; 3324d2e26a3SMauro Carvalho Chehab 3334d2e26a3SMauro Carvalho Chehab flags: 3344d2e26a3SMauro Carvalho Chehab These flags indicate which optional fields are present 3354d2e26a3SMauro Carvalho Chehab in this struct. Currently all fields are mandatory. 3364d2e26a3SMauro Carvalho Chehab 3374d2e26a3SMauro Carvalho Chehab irq: 3384d2e26a3SMauro Carvalho Chehab The IRQ number sent by the AFU. 3394d2e26a3SMauro Carvalho Chehab 3404d2e26a3SMauro Carvalho Chehab reserved field: 3414d2e26a3SMauro Carvalho Chehab For future extensions and padding. 3424d2e26a3SMauro Carvalho Chehab 3434d2e26a3SMauro Carvalho Chehab If the event type is CXL_EVENT_DATA_STORAGE then the event 3444d2e26a3SMauro Carvalho Chehab structure is defined as 3454d2e26a3SMauro Carvalho Chehab 3464d2e26a3SMauro Carvalho Chehab :: 3474d2e26a3SMauro Carvalho Chehab 3484d2e26a3SMauro Carvalho Chehab struct cxl_event_data_storage { 3494d2e26a3SMauro Carvalho Chehab __u16 flags; 3504d2e26a3SMauro Carvalho Chehab __u16 reserved1; 3514d2e26a3SMauro Carvalho Chehab __u32 reserved2; 3524d2e26a3SMauro Carvalho Chehab __u64 addr; 3534d2e26a3SMauro Carvalho Chehab __u64 dsisr; 3544d2e26a3SMauro Carvalho Chehab __u64 reserved3; 3554d2e26a3SMauro Carvalho Chehab }; 3564d2e26a3SMauro Carvalho Chehab 3574d2e26a3SMauro Carvalho Chehab flags: 3584d2e26a3SMauro Carvalho Chehab These flags indicate which optional fields are present in 3594d2e26a3SMauro Carvalho Chehab this struct. Currently all fields are mandatory. 3604d2e26a3SMauro Carvalho Chehab 3614d2e26a3SMauro Carvalho Chehab address: 3624d2e26a3SMauro Carvalho Chehab The address that the AFU unsuccessfully attempted to 3634d2e26a3SMauro Carvalho Chehab access. Valid accesses will be handled transparently by the 3644d2e26a3SMauro Carvalho Chehab kernel but invalid accesses will generate this event. 3654d2e26a3SMauro Carvalho Chehab 3664d2e26a3SMauro Carvalho Chehab dsisr: 3674d2e26a3SMauro Carvalho Chehab This field gives information on the type of fault. It is a 3684d2e26a3SMauro Carvalho Chehab copy of the DSISR from the PSL hardware when the address 3694d2e26a3SMauro Carvalho Chehab fault occurred. The form of the DSISR is as defined in the 3704d2e26a3SMauro Carvalho Chehab CAIA. 3714d2e26a3SMauro Carvalho Chehab 3724d2e26a3SMauro Carvalho Chehab reserved fields: 3734d2e26a3SMauro Carvalho Chehab For future extensions 3744d2e26a3SMauro Carvalho Chehab 3754d2e26a3SMauro Carvalho Chehab If the event type is CXL_EVENT_AFU_ERROR then the event structure 3764d2e26a3SMauro Carvalho Chehab is defined as 3774d2e26a3SMauro Carvalho Chehab 3784d2e26a3SMauro Carvalho Chehab :: 3794d2e26a3SMauro Carvalho Chehab 3804d2e26a3SMauro Carvalho Chehab struct cxl_event_afu_error { 3814d2e26a3SMauro Carvalho Chehab __u16 flags; 3824d2e26a3SMauro Carvalho Chehab __u16 reserved1; 3834d2e26a3SMauro Carvalho Chehab __u32 reserved2; 3844d2e26a3SMauro Carvalho Chehab __u64 error; 3854d2e26a3SMauro Carvalho Chehab }; 3864d2e26a3SMauro Carvalho Chehab 3874d2e26a3SMauro Carvalho Chehab flags: 3884d2e26a3SMauro Carvalho Chehab These flags indicate which optional fields are present in 3894d2e26a3SMauro Carvalho Chehab this struct. Currently all fields are Mandatory. 3904d2e26a3SMauro Carvalho Chehab 3914d2e26a3SMauro Carvalho Chehab error: 3924d2e26a3SMauro Carvalho Chehab Error status from the AFU. Defined by the AFU. 3934d2e26a3SMauro Carvalho Chehab 3944d2e26a3SMauro Carvalho Chehab reserved fields: 3954d2e26a3SMauro Carvalho Chehab For future extensions and padding 3964d2e26a3SMauro Carvalho Chehab 3974d2e26a3SMauro Carvalho Chehab 3984d2e26a3SMauro Carvalho Chehab2. Card character device (powerVM guest only) 3998f97986cSMauro Carvalho Chehab^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 4004d2e26a3SMauro Carvalho Chehab 4014d2e26a3SMauro Carvalho Chehab In a powerVM guest, an extra character device is created for the 4024d2e26a3SMauro Carvalho Chehab card. The device is only used to write (flash) a new image on the 4034d2e26a3SMauro Carvalho Chehab FPGA accelerator. Once the image is written and verified, the 4044d2e26a3SMauro Carvalho Chehab device tree is updated and the card is reset to reload the updated 4054d2e26a3SMauro Carvalho Chehab image. 4064d2e26a3SMauro Carvalho Chehab 4074d2e26a3SMauro Carvalho Chehabopen 4084d2e26a3SMauro Carvalho Chehab---- 4094d2e26a3SMauro Carvalho Chehab 4104d2e26a3SMauro Carvalho Chehab Opens the device and allocates a file descriptor to be used with 4114d2e26a3SMauro Carvalho Chehab the rest of the API. The device can only be opened once. 4124d2e26a3SMauro Carvalho Chehab 4134d2e26a3SMauro Carvalho Chehabioctl 4144d2e26a3SMauro Carvalho Chehab----- 4154d2e26a3SMauro Carvalho Chehab 4164d2e26a3SMauro Carvalho ChehabCXL_IOCTL_DOWNLOAD_IMAGE / CXL_IOCTL_VALIDATE_IMAGE: 4174d2e26a3SMauro Carvalho Chehab Starts and controls flashing a new FPGA image. Partial 4184d2e26a3SMauro Carvalho Chehab reconfiguration is not supported (yet), so the image must contain 4194d2e26a3SMauro Carvalho Chehab a copy of the PSL and AFU(s). Since an image can be quite large, 4204d2e26a3SMauro Carvalho Chehab the caller may have to iterate, splitting the image in smaller 4214d2e26a3SMauro Carvalho Chehab chunks. 4224d2e26a3SMauro Carvalho Chehab 4234d2e26a3SMauro Carvalho Chehab Takes a pointer to a struct cxl_adapter_image:: 4244d2e26a3SMauro Carvalho Chehab 4254d2e26a3SMauro Carvalho Chehab struct cxl_adapter_image { 4264d2e26a3SMauro Carvalho Chehab __u64 flags; 4274d2e26a3SMauro Carvalho Chehab __u64 data; 4284d2e26a3SMauro Carvalho Chehab __u64 len_data; 4294d2e26a3SMauro Carvalho Chehab __u64 len_image; 4304d2e26a3SMauro Carvalho Chehab __u64 reserved1; 4314d2e26a3SMauro Carvalho Chehab __u64 reserved2; 4324d2e26a3SMauro Carvalho Chehab __u64 reserved3; 4334d2e26a3SMauro Carvalho Chehab __u64 reserved4; 4344d2e26a3SMauro Carvalho Chehab }; 4354d2e26a3SMauro Carvalho Chehab 4364d2e26a3SMauro Carvalho Chehab flags: 4374d2e26a3SMauro Carvalho Chehab These flags indicate which optional fields are present in 4384d2e26a3SMauro Carvalho Chehab this struct. Currently all fields are mandatory. 4394d2e26a3SMauro Carvalho Chehab 4404d2e26a3SMauro Carvalho Chehab data: 4414d2e26a3SMauro Carvalho Chehab Pointer to a buffer with part of the image to write to the 4424d2e26a3SMauro Carvalho Chehab card. 4434d2e26a3SMauro Carvalho Chehab 4444d2e26a3SMauro Carvalho Chehab len_data: 4454d2e26a3SMauro Carvalho Chehab Size of the buffer pointed to by data. 4464d2e26a3SMauro Carvalho Chehab 4474d2e26a3SMauro Carvalho Chehab len_image: 4484d2e26a3SMauro Carvalho Chehab Full size of the image. 4494d2e26a3SMauro Carvalho Chehab 4504d2e26a3SMauro Carvalho Chehab 4514d2e26a3SMauro Carvalho ChehabSysfs Class 4524d2e26a3SMauro Carvalho Chehab=========== 4534d2e26a3SMauro Carvalho Chehab 4544d2e26a3SMauro Carvalho Chehab A cxl sysfs class is added under /sys/class/cxl to facilitate 4554d2e26a3SMauro Carvalho Chehab enumeration and tuning of the accelerators. Its layout is 4564d2e26a3SMauro Carvalho Chehab described in Documentation/ABI/testing/sysfs-class-cxl 4574d2e26a3SMauro Carvalho Chehab 4584d2e26a3SMauro Carvalho Chehab 4594d2e26a3SMauro Carvalho ChehabUdev rules 4604d2e26a3SMauro Carvalho Chehab========== 4614d2e26a3SMauro Carvalho Chehab 4624d2e26a3SMauro Carvalho Chehab The following udev rules could be used to create a symlink to the 4634d2e26a3SMauro Carvalho Chehab most logical chardev to use in any programming mode (afuX.Yd for 4644d2e26a3SMauro Carvalho Chehab dedicated, afuX.Ys for afu directed), since the API is virtually 4654d2e26a3SMauro Carvalho Chehab identical for each:: 4664d2e26a3SMauro Carvalho Chehab 4674d2e26a3SMauro Carvalho Chehab SUBSYSTEM=="cxl", ATTRS{mode}=="dedicated_process", SYMLINK="cxl/%b" 4684d2e26a3SMauro Carvalho Chehab SUBSYSTEM=="cxl", ATTRS{mode}=="afu_directed", \ 4694d2e26a3SMauro Carvalho Chehab KERNEL=="afu[0-9]*.[0-9]*s", SYMLINK="cxl/%b" 470