Documentation/powerpc/cxlflash.rst

9     Accelerator Power Interface), which is available to certain PCIe slots
10     on Power 8 systems. CAPI can be thought of as a special tunneling
12     purpose co-processors which can read or write an application's
13     memory and generate page faults. As a result, the host interface to
19     devices as a PCI device by implementing a virtual PCI host bridge.
29     The CXL Flash Adapter Driver is a kernel module that sits in the
30     SCSI stack as a low level device driver (below the SCSI disk and
31     protocol drivers) for the IBM CXL Flash Adapter. This driver is
35     as described in Documentation/powerpc/cxl.rst.
40         - Any flash device (LUN) can be configured to be accessed as a
41           regular disk device (i.e.: /dev/sdc). This is the default mode.
43         - Any flash device (LUN) can be configured to be accessed from
46           either raw access to the entire LUN (referred to as direct
47           or physical LUN access) or access to a kernel/AFU-mediated
48           partition of the LUN (referred to as virtual LUN access). The
49           segmentation of a disk device into virtual LUNs is assisted
59     directly in each user I/O, but at the minimum is involved in the
60     initial setup before the user application is allowed to send requests
67                      +-------------------------------+
71                      +-------------------------------+
75                      +-------------------------------+
78                      +-------------------------------+
80     This driver configures itself into the SCSI software stack as an
81     adapter driver. The driver is the only entity that is considered a
96     The first mode is called a virtual mode. In this mode a single scsi
98     virtual LUNs. The virtual LUNs may be resized as long as the sum of
99     the sizes of all the virtual LUNs, along with the meta-data associated
102     The second mode is called the physical mode. In this mode a single
104     and the entire space for the LUN is available to the application.
116     space should use the block library, as it abstracts the details of
119     The block library can be thought of as a 'user' of services,
120     implemented as IOCTLs, that are provided by the cxlflash driver
122     mode. While it is not a requirement that applications understand
124     a high-level overview of each supported service (IOCTL) is provided
128     http://github.com/open-power/capiflash
134     Users, such as the block library, that wish to interface with a flash
136     by the cxlflash driver. As these services are implemented as ioctls,
139     descriptor is obtained by opening the device special file associated
141     discovery. As per the location of the cxlflash driver within the
142     SCSI protocol stack, this open is actually not seen by the cxlflash
144     (herein referred to as fd1) that should be used for issuing the
151 ------------------
157     referred to as fd2) that is used by the block library to initiate
159     completion events. This file descriptor is intentionally installed by
161     notification and access in the event of a non-user-initiated close(),
162     such as a killed process. This design point is described in further
168         - These tokens are only valid for the process under which they
173         - These tokens are only valid for the lifetime of the context and
174           the process under which they were created. Once either is
178 	- A valid adapter file descriptor (fd2 >= 0) is only returned on
181 	  do not provide the adapter file descriptor as it was previously
184         - When a context is no longer needed, the user shall detach from
187 	  DK_CXLFLASH_APP_CLOSE_ADAP_FD is present, the application _must_
190 	- When this ioctl returns with a valid fd2 and the return flag
191 	  DK_CXLFLASH_APP_CLOSE_ADAP_FD is present, the application _must_
199         - At any time, a close on fd2 will invalidate the tokens. Applications
204 -----------------------
205     This ioctl is responsible for transitioning the LUN to direct
207     user space on a per-context basis. Additionally, the block size and
210     As mentioned previously, when operating in user space access mode,
211     LUNs may be accessed in whole or in part. Only one mode is allowed
212     at a time and if one mode is active (outstanding references exist),
215     The AFU is configured for direct access from user space by adding an
216     entry to the AFU's resource handle table. The index of the entry is
217     treated as a resource handle that is returned to the user. The user
218     is then able to use the handle to reference the LUN during I/O.
221 ------------------------
222     This ioctl is responsible for transitioning the LUN to virtual mode
224     on a per-context basis. Additionally, the block size and last logical
227     As mentioned previously, when operating in user space access mode,
228     LUNs may be accessed in whole or in part. Only one mode is allowed
229     at a time and if one mode is active (outstanding references exist),
232     The AFU is configured for virtual access from user space by adding
234     is treated as a resource handle that is returned to the user. The
235     user is then able to use the handle to reference the LUN during I/O.
237     By default, the virtual LUN is created with a size of 0. The user
241     option of specifying a size as part of the DK_CXLFLASH_USER_VIRTUAL
242     ioctl, such that when success is returned to the user, the
243     resource handle that is provided is already referencing provisioned
244     storage. This is reflected by the last LBA being a non-zero value.
246     When a LUN is accessible from more than one port, this ioctl will
249     of an I/O error as the LUN can be reached over multiple paths.
252 -----------------------
253     This ioctl is responsible for resizing a previously created virtual
254     LUN and will fail if invoked upon a LUN that is not in virtual
255     mode. Upon success, an updated last LBA is returned to the user
259     The partitioning of virtual LUNs is jointly mediated by the cxlflash
260     driver and the AFU. An allocation table is kept for each LUN that is
264     This ioctl can return -EAGAIN if an AFU sync operation takes too long.
267     it is expected to succeed. If this ioctl fails with -EAGAIN, the user
268     can either retry the operation or treat it as a failure.
271 -------------------
272     This ioctl is responsible for releasing a previously obtained
274     thought of as the inverse of the DK_CXLFLASH_USER_DIRECT or
276     is no longer valid and the entry in the resource handle table is
279     As part of the release process for virtual LUNs, the virtual LUN
280     is first resized to 0 to clear out and free the translation tables
284 ------------------
285     This ioctl is responsible for unregistering a context with the
296 ----------------------
297     This ioctl is responsible for cloning a previously created
302     as the parent, but under a different context.
304     Context sharing across processes is not supported with CXL and
312     The clone itself is fairly simple. The resource handle and lun
319     clone. This is to avoid a stale entry in the file descriptor table of the
322     This ioctl can return -EAGAIN if an AFU sync operation takes too long.
325     it is expected to succeed. If this ioctl fails with -EAGAIN, the user
326     can either retry the operation or treat it as a failure.
329 ------------------
330     This ioctl is used to detect various changes such as the capacity of
332     where the changes affect the application (such as a LUN resize), the
336     changed in response to a check condition. As the user is operating out
339     behavior is to call in to this ioctl, indicating what they want to
341     verifying a LUN change (ie: size different) with sense data is
345 -----------------------
346     This ioctl is used to drive recovery (if such an action is warranted)
348     is re-established upon successful recovery.
351     be reset or is terminating. Users are notified of this error condition
353     architected behavior for a user is to call into this ioctl to recover
355     check if the device is operating normally. If a failure is returned
356     from this ioctl, the user is expected to gracefully clean up their
358     hold is not relinquished. The user may also optionally exit the process
359     at which time the context/resources they held will be freed as part of
368 ----------------------
369     This ioctl is used to switch a LUN from a mode where it is available
370     for file-system access (legacy), to a mode where it is set aside for
371     exclusive user space access (superpipe). In case a LUN is visible
372     across multiple ports and adapters, this ioctl is used to uniquely
379     Each host adapter instance that is supported by the cxlflash driver
391 -------------------------
392     This ioctl is used to create and delete persistent LUNs on cxlflash
393     devices that lack an external LUN management interface. It is only
396     When sufficient space is available, LUNs can be created by specifying
410     To query the LUN geometry of a port, the target port is specified and
411     upon success, the following information is presented:
413         - Maximum number of provisioned LUNs allowed for the port
414         - Current number of provisioned LUNs for the port
415         - Maximum total capacity of provisioned LUNs for the port (4K blocks)
416         - Current total capacity of provisioned LUNs for the port (4K blocks)
422 ---------------------
423     This ioctl is used to debug AFUs by supporting a command pass-through
424     interface. It is only valid when used with AFUs that support the AFU
428     cxlflash and treated as pass-through. For debug commands that do require
430     specify the data transfer direction with respect to the host. There is a
432     are not supported - when errors are experienced with a host read data
433     transfer, the data buffer is not copied back to the user.